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1  EXECUTIVE  SUMMARY 


The  goal  of  this  project  is  to  conduct  research  leading  to  identification  and 
analysis  of  predictive  modeling  capabilities  to  support  behavior  influence 
operations.  The  key  deliverable  is  a  set  of  recommendations  for  mature 
analytical  tools  that  meet  the  needs  of  National  Air  and  Space  Intelligence  Center 
(NASIC)  Behavioral  Influences  Analysis  Division  leaders  and  analysts.  Though 
text  mining  and  social  network  analysis  tools  are  important  for  this  customer,  the 
primary  focus  of  this  effort  was  to  identify  the  best  Bayesian  modeling  capabilities 
for  this  mission  area. 

Project  work  was  executed  by  a  government/contractor  team  of  software, 
intelligence,  modeling,  cognitive  science,  operations  research,  engineering,  and 
psychology  experts  from  NASIC,  AFRL,  and  SRA  International,  Inc.  Project  work 
started  in  August  2003  and  completed  in  December  2003. 

SRA  personnel  executed  this  project  by  dividing  the  work  into  four  tasks: 

•  Task  1:  Develop  capability-based  criteria 

•  Task  2:  Market  research  -  identify  simulations,  models,  tools 

•  Task  3:  Rigorous,  objective  assessment  of  simulations,  models,  tools 

•  Task  4:  Develop  recommendations  regarding  acquisition  of  tools 

Detailed  Criteria,  developed  from  a  thorough  understanding  of  the  mission 
requirements,  fell  into  four  criteria  categories:  General  Features,  Input 
Manipulation,  Performance  Extensibility,  and  Interoperability.  Using  contacts 
identified  by  the  customer,  our  own  knowledge  of  the  market,  and  additional 
investigations  using  keyword  searches  of  relevant  databases  and  the  world-wide- 
web,  the  team  developed  an  extensive  list  of  over  80  candidate  tools.  From  this 
list,  we  selected  a  prioritized  set  for  further  data  collection  and  detailed  analysis. 

The  Analytical  Hierarchy  Process  (AHP)  technique  was  applied  to  evaluate 
and  rank  the  tools.  From  a  performance  standpoint,  Hugin  ranks  as  the  optimal 
Bayesian  tool  among  the  alternatives.  The  next  best  performance  alternatives 
are  BayesiaLab  and  Netica.  From  an  “ease  of  use”  perspective,  Netica  and 
Analytica  rank  highest.  Cost  and  pricing  structures  are  roughly  comparable,  with 
the  notable  exception  of  SIAM,  which  is  available  free  of  licensing  fees  to 
government  organizations. 

Our  conclusion,  based  on  this  analysis,  is  that  several  mature  alternatives 
are  available  to  NASIC.  Investment  decisions  could  be  made  on  the  basis  of  this 
report,  but  a  more  sound  approach  would  be  to  develop  and  run  a  benchmark 
problem  scenario  on  a  demonstration  version  of  the  highly  rated  alternatives. 
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2  BACKGROUND 


2.1  Project  Description 

The  goal  of  this  project  is  to  conduct  research  leading  to  the  identification 
and  analysis  of  predictive  modeling  and  analysis  tools  and  capabilities  to  support 
behavior  influence  operations.  The  key  deliverable  is  a  set  of  recommendations 
for  mature  analytical  tools  that  meet  the  needs  of  National  Air  and  Space 
Intelligence  Center  (NASIC)  Behavior  Influence  Analysis  Division  leaders  and 
analysts. 

Project  work  was  executed  by  a  multi-disciplinary,  government/contractor 
team  of  software,  intelligence,  modeling,  cognitive  science,  operations  research, 
engineering,  and  psychology  experts  from  NASIC,  AFRL,  and  SRA  International, 
Inc.  Project  work  started  in  August  2003  and  completed  in  December  2003. 

The  customer  for  this  work  is  Greg  Jannarone,  Chief  of  the  relatively  new 
Behavioral  Influences  Analysis  Division  at  NASIC  (NASIC/BPB).  Gilbert  G. 
Kuperman,  Principal  Mathematician  in  the  AF  Research  Laboratory’s  Human 
Effectiveness  Directorate,  and  members  of  his  staff  also  participated  in  this  effort. 

Government  personnel  provided  some  valuable  early  guidance  and 
organizational  contacts  to  get  us  started  in  this  effort.  They  provided  a  basic 
cultural/institutional/psychological  framework  for  the  end-state  model,  desired 
capabilities  that  helped  us  articulate  requirements,  thoughts  on  initial  evaluation 
criteria  (e.g.,  easy  to  use,  cheap,  can  access  all  information/data  sources 
needed,  accurate,  reliable,  technically  mature,  risks  are  manageable, 
supports/enhances  key  capabilities,  extensible,  flexible,  interoperable  or  easy  to 
confederate,  supportable,  etc.),  and  finally  some  valuable  leads  on  people  and 
organizations  who  could  help  us  get  started. 

This  project  is  envisioned  as  a  foundation  for  future  work  by  NASIC  analysts. 
They  will  use  the  tools  identified  in  this  study  to  assemble  data  and  build  models 
of  individuals  and  small  groups  of  interest,  ultimately  leading  to  the  identification 
and  exploration  of  predictive  modeling,  simulation,  and  analysis  capabilities  in 
support  of  USAF  (and  potentially  Joint)  behavioral  influence  operations. 

SRA  personnel  executed  this  project  by  dividing  the  work  into  tasks  (listed 
below  and  described  in  more  detail  elsewhere  in  this  report): 

•  Taskl:  Develop  capability-based  criteria  to  assess  predictive 
constructive  simulation,  descriptive  models,  other  analytical  tools 
(review  NASIC  goals  for  products,  end  states,  analytical  processes) 

•  Task  2:  Market  research  -  identify  simulations,  models,  tools 

•  Task  3:  Rigorous,  objective  assessment  of  simulations,  models,  tools 
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•  Task  4:  Develop  recommendations,  regarding  acquisition  of  tools 

2.2  Architecture  Context  for  This  Work 

AFDD  2-5  [10],  the  USAF’s  Doctrine  Document  on  Information  Operations 
states  that  “Influence  Operations  are  an  integral  part  of  modern  aerospace 
strategy,” and  defines  Influence  Operations  as  one  of  three  core  Information 
Operations  capabilities  (the  others  are  Electronic  Warfare  Operations  and 
Network  Warfare  Operations).  According  to  AFDD  2-5  (2003  draft): 

Influence  operations  are  focused  on  affecting  the  perceptions 
and  behaviors  of  people,  leaders,  groups  or  entire  populations.  The 
means  of  influencing  can  be  physical  or  informational.  The 
cognitive  domain  is  composed  of  separate  minds  and  personalities 
and  influenced  by  societal  norms.  The  cognitive  domain  isn't 
homogenous,  continuous  or  even  necessarily  rational. 

2003  Draft  AFDD  2-5,  Chapter  1 ,  Page  3 

According  to  the  CONOP  for  AF  Effects  Based  Operations  [1 1]:  “The  most 
significant  challenge  is  to  link  (trace,  understand,  predict,  assess)  actions 
[alternatives  executed  by  blue  forces]  to  [adversary]  behavioral  outcomes.” 

The  29  July  2003  draft  AF  Psychological  Effects-Based  Operations  (AF 
PEBO)  Architecture  OV-2  (Paragraph  2.4. 5.2)  [12]  defines  one  important  node  in 
the  architecture  as  a  NASIC  Behavior  Influences  Analysis  Branch  chartered  to 
“provide  relevant  intelligence  analysis  and  products  supporting  AF  PEBO  and 
joint  PSYOP  targeting  and  planning.”  The  stated  mission  of  this  organization  is 
to  “conduct  human  target  vulnerability  analysis.”  An  SRA  analysis  of  the  trace  of 
this  NASIC  mission  to  other  elements  of  the  overall  Influence  Operations  and 
PEBO  Architecture  is  provided  as  Attachment  1. 

Modern  intelligence/operations  architectures  will  ultimately  deliver  increased 
capability  to  the  warfighter,  but  only  if  the  architecture  enables  true  integrating 
strategies  among  relevant  functional  experts.  Pre-operation  estimates  of 
adversary  reactions  to  influences  are  certainly  possible.  Individual  and  small 
group  behavior  is  rooted  in  culture,  organizational/institutional  ties,  and  unique, 
“knowable”  psychological  factors.  A  holistic,  fused,  all-source  approach  is 
common  to  all  successful  operations,  and  is  critical  to  connect  cultural, 
institutional,  and  psychological  influence  factors.  However,  the  potential  universe 
of  information  and  possible  relationships  to  characterize  human  targets  of 
interest  is  very  large.  Therefore,  analysts  and  decision  makers  need  tools  to 
assist  them  with  this  mission. 

Ultimate  customers  for  the  capabilities  developed  from  follow-on  efforts  to 
this  project  will  include  PSYOP,  Deception,  HUMINT,  Information  Operations, 
Kinetic  Operations,  and  PEBO  planners/targeteers. 
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2.3  Project  Definition  and  Scope 

2.3.1  Objectives 

Current  Project  Product  Objectives: 

•  Recommendations  regarding  acquisition  of  COTS/GOTS  tool  suite, 
based  on  rigorous,  objective  assessment  of  existing  tools 

•  Include  performance,  cost,  and  usability  criteria 

Customer  Process  Objectives  Supported  By  This  Project: 

•  Support  accurate  probabilistic  prediction  of  behaviors  in  response  to 
input  COAs 

•  Enhance  NASIC  effort  to  support  effects-based  operations 

•  Determine  key  psycho/social  factors  that  influence  targeted  individuals, 
groups 

•  Develop  templates  for  relative  influence  weights/probabilities  of 
influence 

•  Develop  estimative  confidence  levels  for  decision-making  factors 

•  Estimation  process  for  “predicting”  potential  anomalous/irrational 
behavior 

•  Probabilistic  model  of  likely  decision-making  behavior 

2.3.2  Project  Framework 

SRA  planners  felt  it  was  necessary  to  fully  understand  and  appreciate  the 
elements  of  the  capability  envisioned  to  meet  the  needs  of  our  customer.  One 
way  to  capture  this  understanding  is  by  building  some  alternative  frameworks  for 
the  capabilities.  Figure  1  shows  two  alternative  views  of  an  implementing 
framework  for  the  NASIC  vision. 


*\ 
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Alternative  model 


(  Objectives  ) 


Desired  Effect 


ROE 

Capability 

Constraints 

Tendencies 


Should  Equal 


[Predicted  Effect 


Figure  1.  Two  alternative  Project  Capability  Frameworks 

What  are  we  trying  to  represent  within  either  of  these  frameworks?  First, 
there  is  a  need  to  characterize  target  (in  this  case  individual  human  beings  or 
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groups  of  human  beings)  vulnerabilities:  what  target  characteristics  or  attributes 
can  planners  and  analysts  leverage  and  exploit?  In  conjunction  with  this,  there  is 
a  related  but  distinct  need  to  characterize  target  susceptibilities:  can  the  target 
behave  in  the  way  we  expect  or  desire?  Is  there  a  predictable  path  to  exploit  the 
vulnerability?  Finally,  there  is  the  need  to  determine  accessibility:  can  our  target 
of  interest  be  reached  (physically  or  other)  to  use  the  susceptibility  and  exploit 
the  vulnerability? 

Within  the  layers/filters  of  the  frameworks,  note  there  are  categories  of  target 
attributes  that  may  be  exploitable: 

•  Psychological  attributes  -  What  can  we  know  (emotions,  ambitions, 
motivations,  goals,  needs)? 

•  Institutional  attributes  -  What  happens  when  we  sever  or  influence  key 
links? 

•  Cultural  attributes  -  Which  of  the  numerous  candidate  cultural 
attributes  do  we  really  care  about?  Which  are  the  best  contributors  to 
a  prediction  of  target  behavior? 

What  we’re  really  looking  for  in  an  influence  operations  framework,  then,  are 
influence  links  that  originate  in  culture,  pass  through  organizations  and 
institutions,  into  the  minds  of  individuals  and  groups  -  thread(s)  that  reliably  link 
the  three  planes  of  influence  can  put  us  on  the  path  of  a  predictive  capability. 

Any  selected  tool  or  suite  of  tools  must  provide  the  environment  for  implementing 
this  capability. 

SRA’s  investigation  identified  three  categories  of  tools  that  are  needed  to 
implement  the  analysis  framework  alternatives  illustrated  in  Figure  1 .  These 
categories  are: 

•  Text/Data  Mining  capabilities 

•  Social  Network  Analysis  capabilities 

•  Modeling  capabilities 

Though  this  study  took  all  three  categories  into  account,  most  (90%)  of  the 
focus  for  the  SRA  team’s  effort  was  on  identifying  and  analyzing  modeling 
capabilities.  This  decision  was  made,  with  the  full  support  of  the  customer,  to 
ensure  the  main  project  objectives  could  be  achieved  within  resource  constraints. 
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3  REQUIREMENTS 


3.1  Vision 

NASIC  Behavioral  Influences  Analysis  Division  Vision:  Identify  influences  that 
can  condition,  alter  perceptions,  get  the  attention  of  and  (ultimately)  effect  the 
behavior  of  adversaries  (individuals,  small  decision-making  groups). 

3.2  Mission 

Division  Mission:  Help  the  AF  build  a  concept  of  operations  that  integrates 
intelligence  and  information  operations/warfare  communities  as  an  “influence 
operations  capability/system”  (that  links  actions  and  behaviors). 

3.3  Critical  Immediate  Need 

Critical  immediate  need:  Identify  modeling,  simulation,  and  analysis  tools  and 
capabilities  for  pre-operation  estimation  of  behavioral  influence  operation  option 
effects.  Need  is  urgent  for  tools/capabilities  to  help  analysts  establish  a  solid 
analytical  foundation  for  this  emerging  mission  area. 

3.4  NASIC  Analyst  Input 

On  19  Sep  03,  an  SRA  team  interviewed  four  NASIC/BPB  Branch  Chiefs 
and  the  BPB  Functional  Team  Lead  [13].  The  following  paragraphs  summarize 
our  interviews.  This  is  presented  because  it  helped  form  a  critical  link  between 
high  level  requirements  presented  in  the  PEBO  Architecture  (see  Paragraph  2.2) 
and  the  criteria  we  developed  for  this  study.  Analyst  questions  are  provided  as 
Attachment  2. 

SRA’s  main  goal  was  to  capture  the  analysts’  focus,  expectations,  and 
priorities  as  it  pertains  to  the  current  project.  Following  are  some  top  level 
observations: 

•  Analysts  with  different  areas  of  responsibility  will  have  some 
overlapping  focus  areas,  needs,  expectations,  and  priorities,  but  will 
also  have  unique  needs  associated  with  unique  problems  in  their 
assigned  area. 

•  Some  will  focus  on  small  groups  and  individuals  in  the  near  term. 

•  Some  will  rely  more  on  organizational  and  cultural  information  because 
detailed  information  on  individuals  will  be  hard  to  come  by. 

•  Analyst  skill  set  will  range  from  relatively  new  analysts  to  20-year 
analysis  veterans  with  knowledge  and  expertise  in  multiple  “I NTs.” 

•  Analysts  will  support  a  range  of  customers  (intelligence,  planners, 
operators,  targeteers,  IWF  personnel),  in  deliberate  planning  and  crisis 
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action  modes,  primarily  at  the  operations  and  strategic  levels  of 
conflict. 

•  Typical  products  will  include  target  profiles,  target  folder  supplements, 
target  system  analyses,  short-suspense  RFIs,  pre-positioned  O-Plan 
products,  and  vulnerability  assessments. 

•  Available  information  should  be  the  LIMFAC,  not  the  analytic  tool  set. 

•  Analysts  are  very  concerned  about  data  overload  and  data 
organization,  not  as  much  about  target  models  at  this  point  in  time. 
Their  priorities  focused  on  user-friendly,  data  mining  (associative 
pattern  mining)  and  relational  database  tools  first,  model  building  and 
predictive  tools  second. 

•  Analysts  do  not  expect  an  analytic  tool  set  to  replace  their  knowledge 
and  expertise.  They  expect  that  they  and  their  functional  experts  will 
provide  the  knowledge  needed  for  the  modeling  capability.  Aids  to 
help  remember  and  organize  knowledge  would  be  welcome  capability 
additions. 

•  Models  will  need  to  handle  at  least  300  nodes,  perhaps  up  to  3000 
nodes. 

It  became  evident  after  these  interviews  that  the  SRA  team  needed  to  focus 
on  three  technology  classes  to  provide  the  overall  analytic  capability  needed  by 
BPB  analysts:  text/data  mining  tools  (to  group  and  summarize  documents),  link 
analysis  tools  (e.g.,  relational  database  mining  tools  to  find  links  between 
individuals  and  groups/events/attributes),  and  modeling  tools  (to  build  belief 
nets).  Some  of  the  specific  needs/desires  for  an  analytical  capability  articulated 
by  the  analysts  include: 

•  Flexibility  to  adapt  to  needs  of  individual  analysts 

•  Ability  to  grow  in  size/sophistication  as  information  set  matures 

•  Ability  to  search  by  time  relationships,  group  associations,  family 
relationships,  geographic  proximity,  etc. 

•  Access  classified  (multi-INT)  and  open  source  information 

•  Efficient  pulling  and  loading  (text,  photos,  e-files,  paper,  web  pages, 
etc.) 

•  Capability  to  provide  reminders  or  triggers  on  saved  data 

•  Efficient  organization  of  information  so  it  can  be  easily  found  when 
needed 

•  Liked  the  SIAM  logic  train  (how  it  maintains  track  to  source) 

•  Want  tools  to  help  them  “lay  the  evidence  out” 
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•  Capability  to  assign  probabilities  of  given  events  (not  necessarily  to 
have  modeling  tool  assign  the  probability) 

•  Spider  diagrams  will  aid  social  network  analysis 

•  Want  interoperability  with  other  agencies’  tools  (e.g.,  Visual  Links, 
Analysts  Notebook) 

•  Willing  to  be  trained,  but  the  less  training  needed,  the  better 

•  Data/text  mining  and  relational  data  capabilities  will  populate  a 
hierarchy  of  cultural,  organizational,  individual  events 

•  Want  node/link  structure,  weights  easily  tailored  to  a  given  target 

•  Analysts  have  techniques  to  search  for  (and  acquire)  what  they  “know 
they  don’t  know;”  analysts  would  love  to  have  aids  to  help  them  with 
what  they  “don’t  know  they  don’t  know” 

•  Volume  of  information  can  be  overwhelming  to  an  analyst;  they  need 
help  to  sort,  organize  and  prioritize  ~  1300  messages  per  day 
(classified/unclassified  sources)  and  improve  workload  efficiency 

•  Understanding  bias  associated  with  sources  is  a  very  important 
element  of  the  foundation  of  a  predictive  capability 

•  Text  is  important  to  everyone,  visual  information  (e.g.,  pictures)  is 
critical  for  some  due  to  the  shortage  of  useable  text  information 

It  also  became  evident  that  there  are  some  things  the  analysts  will  not  be 
expecting  from  an  initial  analytical  tool  set: 

•  Will  not  be  likely  to  trust  search/data  mining  tools  to  find  all  relevant 
relationships  -  analyst  will  still  play  a  major  role  in  reviewing, 
organizing,  and  making  inferences  from  the  available  data 

•  Multi-media  will  be  useful,  but  is  not  a  priority  now 

Consistent  priorities  for  the  analysts  are,  in  this  order,  user  friendly 
sort/organization  aids,  relational  database,  links/nodes  model  structure,  Bayesian 
nets  for  probabilistic  assessments. 

These  interviews  were  very  valuable  to  the  SRA  team.  This  information 
reinforced  and  clarified  the  user  requirements,  and  facilitated  our  completion  of 
project  Task  1. 

3.5  Final  Project  Requirements  Trace 

Attachment  1  presents  a  final  requirements  trace  from  Influence  Operations 
doctrine  and  architecture  (requirements)  to  the  criteria  we  used  to  evaluate 
candidate  tools  and  capabilities.  This  trace  ensures  that  the  criteria  used  to 
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evaluate  promising  tools  are  measuring  attributes  that  contribute  to  the 
accomplishment  of  the  vision,  mission,  and  requirements  of  NASIC/BPB. 

4  TASK  EXECUTION 


4.1  Methodology 

The  basic  methodology  used  to  execute  this  project  is  described  in  the 
following  steps: 

•  Document/present  a  project  plan 

•  Ensure  requirements  are  well  understood 

•  Map  requirements  to  criteria  (emphasis  on  modeling  tools) 

•  Establish  a  list  of  candidate  tools  by  conducting  a  market  survey 

•  Develop  questionnaire  tied  to  the  evaluation  criteria 

•  Phone  or  personal  contact  with  vendors,  followed  up  with  a  request  to 
complete  the  questionnaire 

•  Enter  data  from  completed  questionnaires  (follow  up  as  necessary) 
into  a  spreadsheet  format 

•  Analyze  data 

•  Report  results  to  customer 

The  project  methodology  was  executed  in  phases  as  described  below: 

•  Phase  1:  Detailed  planning 

•  Review  available  guidance  from  customer 

•  Contact  people  who  understand  mission  requirements 

•  Develop  information  collection  strategy  &  methodology 

•  Assign  initial  team  roles,  responsibilities 

•  Build  a  detailed  project  schedule 

•  Develop  an  initial  capability  and  criteria  list 

•  Present  project  plan  to  customer  for  feedback 

•  Phase  2  (Tasks  1  and  2):  Refine  capabilities/criteria,  complete  market 
research 

•  Identify  project/product  needs,  “alterables”,  constraints 

•  Define  a  product  framework 

•  Identify  functions  that  are  needed  within  the  framework 
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•  Identify  technology  categories  needed  to  implement  the  functions 

•  Develop  criteria  to  evaluate  tools,  technologies 

•  Effects-based  criteria  (ability  to  do  the  job) 

•  Other  criteria  (ease  of  use,  cost,  supportability,  etc.) 

•  Phase  3  (Tasks  3  and  4):  Collect  data,  perform  assessment  on  a 
prioritized  subset  of  the  identified  tools 

•  Identify  existing  tools  that  implement  required  technologies 

•  Evaluate  tools  against  approved  criteria 

•  Phase  4:  Final  report  writing,  recommendations 

•  Recommend  right  mix  of  tools 

•  Immediate  capability  expectations 

4.2  TASK  1:  Develop  Criteria 

Task:  Develop  capability-based  criteria  used  to  assess  “predictive  constructive 
simulation,  descriptive  models,  and  other  analytical  tools”  (review  NASIC  goals 
for  products,  end  states,  analytical  processes). 

Task  1  Deliverable:  Spreadsheet  detailing  and  organizing  the  project  evaluation 
criteria  (Attachment  3). 

Using  the  Requirements  Trace  described  in  Section  3  and  presented  in 
Attachment  1 ,  the  team  outlined  a  basic  structure  of  needs  (important 
characteristics  that  are  “must-haves”  for  the  project),  “alterables”  (criteria  that 
could  be  compromised  and  adjusted  to  fit  the  structure  of  the  project),  and 
constraints  (attributes  that  would  limit  project  scope).  These  needs,  alterables, 
and  constraints  are  presented  below: 

Needs 

•  Supports  target  audience  development 

•  Vulnerabilities,  susceptibilities,  protections 

•  Supports  identification  of  a  range  of  influence  factors 

•  Can  assign  weights  to  influence  factors  (culture,  institution, 
psychological) 

•  Connects  influence  factors  with  candidate  audience  behaviors 
(links/nodes) 

•  Relational  database  handling/management/processing 

•  Can  access  all  data  sources  needed  (U  through  TS) 
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S I PRN  ET -compatible 
Reduces  analyst  workload 


“Alterables” 

•  Easy  to  use  (minimize  training  requirements) 

•  Low  cost 

•  Accurate 

•  Reliable 

•  Mature 

•  Risk  is  manageable 

•  Extensible  (growth  capacity) 

•  Flexible 

•  Interoperable  or  easy  to  confederate 

•  Supportable 

•  Source  code  accessible  or  stable  API 

Constraints 

•  Bandwidth  (text,  relational  database  inputs) 

•  Low  level  of  contractor  support  needed 

•  Don’t  want  to  use  as  stand-alone  tool  (open  arch,  common  formats) 
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Comparing  the  needs,  alterables,  and  constraints  with  the  model  framework 
described  previously,  it  became  clear  that  certain  functions  would  be  required  to 
satisfy  the  basic  design  of  an  Influence  Operations  analytical  capability.  These 
functions  are  illustrated  in  Figure  2. 

Figure  2  describes  three  major  functions:  input,  manipulation,  and  access. 
Within  these  functions  are  sub-functions  (e.g.,  ‘publish,  subscribe,  transform, 
query,’  etc.).  Implementing  these  sub-functions  will  require  specific  technologies. 
Identifying  and  listing  these  technologies  contributes  to  another  level  of 
understanding  with  which  to  build  evaluation  criteria.  An  initial  cut  (developed  in 
the  early  phases  of  the  criteria  development  phase)  on  the  specific  technologies 
needed  to  implement  Influence  Operations  framework  functions  is  presented 
below: 

INPUT  Technologies: 

•  Identification  &  authentication:  Source  certificates,  Secure  ID,  Source 
availability,  Source  discovery,  Source  ID 

•  Access  and  translation:  Multimedia  data  capture,  Wrapper  technology, 
Heterogeneous  data  integration,  Transformation  techniques,  Capture 
plan  data,  Meeting  transcription 

•  Upstream  information:  Tagging  techniques,  Source  characterization, 
Capturing  user  intent,  Pedigree  capture  (source  processing) 

•  Categorization:  Task/data  relevance,  Ontologies  &  taxonomies, 
Expectation  driven  change  detection 

MANIPULATION  Technologies: 

•  Storage:  Multimedia  storage,  Resource  distribution  management, 
Multilevel  secure  storage,  Seamless  access  to  tertiary  storage.  High- 
performance  computing,  Backup  and  recovery,  High-density  mass 
storage,  Data  warehousing 

•  Extraction:  Access  control,  Agent  technologies,  Intelligent  push 
technologies,  Intent  inferencing,  Dynamic  access  control,  Information 
usage  analysis,  User  modeling 

•  Decision  Support:  Structuring,  Advice,  Uncertainty  portrayal,  Tradeoff 
management 

•  Aggregation  and  Fusion:  Video  compression,  Wrapping  legacy 
systems,  Information  synchronization,  Geospatial  and  temporal 
indexing,  Object  extraction  for  compression,  Meta-data  language,  Data 
fusion,  Information  life  cycle,  Database  mediation,  Rapid  knowledge 
formation 
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•  Accessing:  Collaboration  technologies,  Self-healing  networks,  Multi¬ 
level  access  (e.g.,  security),  Parallel  access  for  speed 

•  Labeling:  Uncertainty,  Domain-specific  taxonomies,  ontologies 

•  Understanding:  Dynamic  situation  modeling,  Sensitivity  analysis 

INFORMATION  ACCESS  Technologies: 

•  Routing:  Internet,  ATM  switching,  broadcast  technologies,  etc., 
Dynamic  bandwidth  management 

•  Transmission:  Assured  delivery,  Non-repudiation 

•  Perception:  3-D  Visualization,  Natural  language,  Non-traditional 
senses,  Drilldown,  3-D  audio,  Tailoring 

•  Protection:  Encryption,  Recall  of  inaccurate  information 

•  User  modeling:  Info  needs  models,  Dialog  management,  Context 
understanding,  Intent  inferencing 

•  Communication:  Conversational  query,  Speech,  Natural  language, 
Annotation,  Domain-specific  gesturing 

•  Collaboration:  Sharing,  Advanced  white-boarding,  Domain-specific 
workflow  management,  Mixed  initiatives,  Facilitation 

As  an  example  of  how  a  listing  of  technologies  can  contribute  to  an 
understanding  of  the  criteria  needed  for  this  analysis,  consider  a  Decision 
Support  System  (DSS).  DSS  are  computer-aided  tools  that  emulate  the 
reasoning  process  of  a  human  expert  making  decisions.  DSS  are  currently 
employed  in  the  fields  of  business,  medicine,  law,  and  environmental  sciences. 
For  this  project,  tools  are  needed  that  support  decision  making.  A  DSS 
framework  is  designed  to  analyze  “what-if  scenarios  and  help  make  decisions 
that  involve  multi-criteria  inputs. 

DSS  are  classified  into  categories  based  on  the  services  and  features 
supported.  For  purposes  of  this  project,  SRA  considered  knowledge-driven  DSS 
and  model-driven  DSS.  Knowledge-driven  DSS  are  pre-programmed  with 
specialized-problem  expertise.  They  contain  knowledge  about  a  specific  domain, 
and  use  this  knowledge  to  aid  the  decision  process  of  the  user.  The  benefit  of 
this  type  of  DSS  is  that  domain  experts  can  “train”  the  tool  in  the  initial  stages 
and  prepare  it  so  that  a  user  need  not  be  a  domain  expert.  The  drawbacks  are 
that  training  of  the  tool  is  cumbersome,  and  it  may  not  be  very  adaptive  to 
changing  scenarios. 

Model-driven  DSS  (SIAM™  software  is  an  example)  use  statistical  models  to 
support  decision-making.  These  DSS  technologies  rely  on  the  data  provided  by 
the  user  (decision  tree,  weighting  of  the  nodes)  for  analysis.  On  the  positive  side, 
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no  overhead  of  training  the  tools  is  required,  and  these  DSS  tend  to  be  adaptive 
to  changing  scenarios.  The  main  drawback  is  that  any  given  user  must  be  a 
domain  expert  to  apply  the  tool  effectively. 

While  this  is  just  one  example  of  how  a  knowledge  of  the  technologies  can 
be  applied  to  develop  criteria  for  tool  analysis,  one  trend  should  be  made  clear: 

All  of  the  technologies  evaluated  for  this  project  will  have 
advantages  and  disadvantages,  so  this  meant  it  was  critically 
important  to  ensure  customer  needs  were  well  understood  and 
that  the  customer  was  involved  in  all  aspects  of  evaluation  and 
selection  criteria  development,  prioritization,  and  weighting. 

As  a  further  illustration  of  how  we  used  an  understanding  of  relevant 
technologies  to  select  and  prioritize  criteria,  refer  to  Table  1.  This  table  presents, 
for  two  key  categories  of  tools  (Link-Node  Analysis  and  Decision  Support 
Systems),  potential  techniques  used,  concepts  and  theory  applied,  concerns,  and 
a  candidate  set  of  criteria  that  could  be  used  to  evaluate  the  tool. 


Table  1.  Illustration  of  Understanding  Technologies 


Categories 

Link  -  Node 

Analysis 

Decision  Support  Systems  (DSS) 

Technique  Used 

Link  Analysis  -  Finds 

and  represents 
patterns  between 
different  variables  in 
the  data  sets 

Model  -  driven 
(Decision  Analysis  Tools) 

Knowledge  -  driven 

Concepts/Theory 

Applied 

Data  Mining? 

Statistical  approach 
(Bayesian  Networks) 

Case  Based 

Reasoning  (CBR) 

Rule  Based 

Reasoning  (RBR) 

-  "Learning”  concepts 

Concerns 

How  do  the  tools 

decide  which 
variables  to  use  to 
discover  patterns? 
Does  the  user  specify 
these  variables? 

What  other 
information  does  the 
system  need? 

What  is  the  output: 
Sensitivity  Analysis  or 
Impact  Analysis 

For  CBR 

Similarity  Function 
Indexing/Retrieval 
technique 

For  RBR 

Experts  define  rules 

Use  data-mining  to 
generate  rules 

Criteria 

Interactivity  with 

the  user 

Representation 

aiding 

Ability  to  interface 
with  multiple 
databases 

Interactivity  with  the 
user 

Collaborative  support 
Qualitative  vs. 
Quantitative  weights? 
Assessment  utilities 
used 

Interactivity  with  the 
user 

Predictive  capability? 
Evaluate  data-mining 
tools? 
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There  are  some  key  terms  that  will  keep  coming  up  in  this  report  and  in  the 
attachments.  They  include: 

•  Probability  Theory 

•  Decision/Utility  Theory 

•  Bayesian/Belief/Causal/Probability  Nets 

•  Influence  Diagrams 

•  Decision  T rees 

•  Criteria  model 

Definitions  of  these  terms  and  a  tutorial  briefing  are  provided  as  Attachment 
4.  This  information  can  also  be  found  in  the  Project  Glossary  (Attachment  1 1 ). 

After  completion  of  our  requirements  analysis,  breaking  down  the  functions, 
sub-functions  and  technologies  within  the  project  framework,  and  in-depth 
discussions  with  our  customers,  the  SRA  team  established  the  criteria  to 
evaluate  candidate  modeling  tools.  Starting  at  a  macro  level,  we  sought 
capabilities  that  fit  the  following  criteria: 

•  Bayesian-based/inferential  design,  to  permit  probabilistic,  or  levels  of 
confidence,  outputs 

•  Analyst-friendly  and  (relatively)  quickly  trainable  operation 

•  Links  and  nodes  derivation  and  representation,  especially  for 
“weighting”  of  the  (influence)  links  and  (individual/group)  nodes 

•  Relational  data/knowledge  base  utilization  and  applications 
functionality 

Detailed  Criteria,  developed  from  a  thorough  understanding  of  the  mission 
requirements  fell  into  four  criteria  categories: 

•  General  Features 

•  Input  Manipulation 

•  Performance  Extensibility 

•  Interoperability 

General  Features  included  criteria  such  as  ease  of  use  (which  was,  in  turn, 
analyzed  in  further  detail  with  more  detailed,  tailored  criteria),  cost,  maturity, 
targeted  industries,  major  clients,  type  of  operating  system,  recommended 
hardware,  and  architecture. 
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Input  Manipulation  criteria  investigated  type  of  network  support,  model 
building  capability,  type  of  graph  support,  conditional  probability  table 
specification,  model  validation,  inference  algorithm  used,  analysis  techniques, 
and  ability  to  limit  computational  complexity. 

Performance  and  Extensibility  criteria  included  the  language  used  for 
development,  availability  of  source  code  and  API,  whether  a  benchmark  was 
used  to  evaluate  tool  performance,  maximum  number  of  nodes  supported,  and 
error  recovery  support. 

Finally,  the  Interoperability  category  looked  at  criteria  such  as  ability  to 
interface  with  databases,  formats  used  to  save  models,  ability  to  export  models 
and  analysis  results  to  other  applications,  use  of  open  architecture  standards, 
and  capability  for  group  collaboration. 

The  complete  set  of  evaluation  criteria  and  results  from  surveys  of  selected 
vendor  capabilities  are  captured  in  Criteria/Data  Capture  Sheets,  a  set  of 
worksheets  presented  in  Attachment  3. 

4.3  TASK  2:  Market  Survey 

Task:  Compile  a  comprehensive  list  (within  resource  constraints)  of  capabilities 
that  could  satisfy  some  or  all  of  the  modeling  requirements  of  this  project. 

Task  2  Deliverables:  Excel  spreadsheet  listing  all  candidate  tools  with  notes  on 
selection  and  elimination  criteria;  list  of  contacts  and  sources  of  information 
(Attachment  3);  briefing  on  how  we  prioritized  candidates  for  detailed  data 
collection  and  assessment  (Attachment  5). 

Using  contacts  identified  by  the  customer,  our  knowledge  of  the  market,  and 
some  additional  investigations  using  keyword  searches  of  relevant  databases 
and  the  world-wide-web,  the  SRA  team  developed  an  extensive  list  of  candidate 
tools.  This  list  included  over  80  candidates  -  about  fifty  of  these  were  judged  to 
be  too  much  in  the  research  phase  (too  immature)  to  be  reliably  evaluated.  The 
remaining  (30+)  tools  were  evaluated  through  another  screen  -  whether  they 
were  directly  applicable  to  the  customer’s  problem  domain. 

The  SRA  team  decided  to  group  the  tools  based  on  the  underlying  technique 
used  for  decision  modeling.  The  tools  were  categorized  into  four  major 
categories:  Bayesian  Networks,  Influence  Networks,  Decision  Trees  and  Criteria 
Modeling  (i.e.,  building  hierarchies  of  criteria  and  ranking  them  using  AHP  or 
SMART  techniques). 

The  following  criteria  helped  prioritize  the  class  of  tools  we  would  carry  into 
the  next  task/phase  (Task  4): 
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IMPORTANT:  Tools  that  support  Bayesian  and  Influence  nets,  permit 
manual  construction  of  nets,  and  support  evidence  entry  and  belief 
updates  (diagnostic  inference). 

NOT  IMPORTANT:  Tools  that  are  add-ins  to  other  applications  (ex: 
spread  sheet  add  in),  tools  that  rely  on  either  decision  tree  technique  or 
Multi-attribute  Utility  Theory  (MAUT)  (the  problem  domain  is  too  big  to  be 
represented  as  one  of  the  above),  or  tools  that  provide  data  mining 
capabilities  given  large  amounts  of  data  (NASIC  analysts  may  not  have 
large  amounts  of  data  in  the  early  stages  of  implementation). 

Tools  that  used  Decision  Trees  were  not  considered  because  we  wanted 
tools  that  represented  decision  models  compactly.  Decision  Trees  require  explicit 
representation  of  every  possible  alternative  and  this  would  soon  result  in  a  model 
that  is  not  manageable  in  size.  Criteria  Modeling  tools  were  not  considered,  as 
they  do  not  allow  users  to  represent  uncertainties  of  how  one  variable  influences 
another  variable.  Also,  the  requirement  of  a  strict  ordering  among  the  criteria 
made  Criteria  Modeling  not  applicable  to  the  problem  domain. 

We  considered  tools  that  used  Bayesian  Networks  or  Influence  Networks  for 
decision  modeling  as  they  allow  for  a  compact  representation  of  the  model  and 
probabilistic  representation  of  uncertainties  of  the  amount  of  influence  one 
variable  has  on  the  other.  For  a  detailed  description  of  Bayesian  Networks  and 
Influence  Networks,  refer  to  Reference  [4]  (at  the  end  of  Section  4). 

Figure  3  outlines  the  decision  process  explained  above  and  lists  the  tools 
considered: 


JEMNA 

SIAM 

WAE 

SAMPLE 

Figure  3.  Market  Survey  Screening  Process 
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4.4  TASK  3:  Data  Collection 

Task:  Based  on  criteria  from  Task  1 ,  the  SRA  team  developed  a  questionnaire 
(Attachment  6)  that  asked  vendors  of  the  “chosen”  tools  (from  Task  2)  to 
characterize  their  product  in  terms  of  our  evaluation  criteria.  Questionnaire 
responses  (Attachment  7)  were  mapped  to  the  Criteria/Data  Capture  Sheet 
(Attachment  3).  This  information  was  then  used  as  input  to  the  information 
analysis  task/phase  of  the  project. 

Task  3  Deliverables:  Raw  data  collected  from  candidate  tool  vendors 
(Attachment  7);  Excel  spreadsheet  summarizing  results  from  surveys 
(Attachment  3). 

4.5  TASK  4:  Data  Analysis 

Task:  Rigorous,  objective  assessment  of  simulations,  models,  tools  (obtain 
demo  SW/models/tools  and  evaluate  against  criteria  from  Task  1).  Assess  how 
application  of  tools  will  support  behavioral  influence  operations. 

Task  4  Deliverables:  Output  from  InfoHarvest  Criterium  DecisionPlus  analytical 
tool  in  Excel  format;  report  on  results  of  analysis 

4.5.1  Multi-Criteria  Decision  Analysis: 

After  collecting  information  on  the  tools  (using  the  questionnaire),  the 
Analytical  Hierarchy  Process  (AHP)  technique  was  applied  to  evaluate  and  rank 
the  Bayesian  tools.  AHP  is  a  Multi  Criteria  Decision  Analysis  (MCDA)  technique 
where  a  hierarchy  of  criteria  is  developed.  At  the  top  level  of  the  hierarchy  is  the 
final  goal  (which  in  this  case  was  the  best  Bayesian  tool  satisfying  most  of  our 
criteria).  The  next  level  of  hierarchy  consists  of  criteria  directly  affecting  the  goal. 
The  subsequent  levels  of  the  hierarchy  consist  of  subcriteria  affecting  the  criteria 
above  them.  The  last  level  of  the  hierarchy  consists  of  the  alternatives  that  are 
being  evaluated  against  the  criteria.  This  hierarchy  represents  the  decision  model 
used  to  evaluate  the  alternatives. 

Once  the  model  is  constructed,  the  next  step  is  to  assign  weights  to  all  the 
elements  (criteria  and  alternatives)  in  the  model.  The  weight  of  an  element 
represents  the  amount  of  contribution/influence  that  element  has  towards  the 
criteria  above  it  when  compared  to  the  other  elements  in  its  level. 

For  our  analysis,  InfoHarvest  Criterium ®  DecisionPlus®  3.0  (CDP)  was  used 
to  determine  the  best  Bayesian  tool  among  all  the  alternatives.  CDP  is  a  decision 
analysis  tool  that  supports  AHP  and  other  MCDA  techniques  to  help  aid  in 
decision  making.  CDP  provides  a  GUI  to  build  the  hierarchies;  the  user  develops 
the  decision  hierarchy  and  then,  using  the  AHP  or  SMART  analysis  technique, 
ranks  the  elements  in  the  hierarchy.  A  scale  of  five  ranking  gradations  is 
supported  by  the  tool  to  assign  ranks  to  the  elements.  Ranks  can  be  assigned 
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numerically  or  verbally.  The  following  table  (Table  2)  summarizes  the  ranking 
scale  available  for  direct  AHP  ranking  in  CDP. 


Table  2.  AHP  Ranking  Scale  in  CDP 


Numerical 

Verbal 

100 

Critical 

75 

Very 

Important 

50 

Important 

25 

Unimportant 

0 

Trivial 

4.5.2  Criteria  Selection  &  Ranking: 

The  criteria  selected  were  based  on  the  questionnaire.  Questions  that  every 
tool  answered  similarly  (e.g.,  Does  the  tool  provide  GUI  support  for  model 
building?)  or  those  that  had  responses  that  were  not  quantifiable  (e.g.,  Who  are 
the  tool’s  major  clients?)  were  not  considered  as  part  of  our  criteria.  Subsets  of 
questions  from  the  questionnaire  were  selected  as  criteria  for  every  category  in 
the  questionnaire. 

The  decision  model  for  choosing  an  optimal  Bayesian  tool  was  developed  as 
a  two  stage  model.  The  first  stage  involved  developing  decision  models  for  every 
category  in  the  questionnaire  and  ranking  the  alternatives  within  each  category. 
These  decision  models  were  called  the  sub-goal  models.  The  final  model  was 
developed  by  grouping  the  four  sub-goal  models  together  as  criteria  for  the  final 
goal. 

Once  the  sub-goal  and  final  goal  models  were  built,  input  from  the  customer 
was  taken  to  assign  preference  ordering  of  the  criteria  and  subcriteria  within  all 
the  decision  models.  These  preference  orderings  were  then  translated  into  the 
ranking  scale  provided  by  CDP  3.0. 

The  following  tables  (Tables  3-7)  summarize,  for  every  category,  the 
questions  that  were  considered  as  criteria/subcriteria  and  their  associated 
weights.  Every  table  is  followed  by  a  CDP  hierarchy  diagram  of  the  table  (Figures 
4-8),  showing  the  normalized  relative  weights  of  criteria  and  appropriate  levels  of 
subcriteria. 
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Category  -  General  Features 


Table  3.  General  Features  Criteria  and  Relative  Weights 


Criteria 

Rel 

Wt 

Subcriteria 

Rel 

Wt 

Questionnaire  questions 
used  from  General 

Features  category 

Maturity 

2 

>5  years 

1 

Question  2.  Introduced 
a  subcriteria  of  less 
than  or  greater  than  5 
years 

<5  years 

2 

OS Support 

3 

MultipleOS  - 
Win,  Linux, 
(and/or) 

Mac 

1 

Question  5. 

Architecture 

1 

Client- 

Server 

1 

Question  7a 

^10.333  Maturity  k; - 

- m-222  >5  vrs  1 

ll.OOO  GeneralFeatures 

^|0.1 11  <5  yrs 

C^^O.222  OS  Support  | - 

- 10.222  MultipleOS  | 

n0.444  Architecture  | - 

- 10.444  Client  Server  | 

Figure  4.  Normalized  relative  weights  for  General  Features  criteria 
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Category.  -  Ingut  Manipulation 

Table  4.  Input  Manipulation  Criteria  and  Relative  Weights 


Criteria 

Rel  Wt 

Subcriteria 

Rel  Wt 

Questionnaire  questions 
used  from  Input 

Manipulation  category 

Decision  Node  Support 

3 

Question  1b 

Analysis  Technique 

1 

Question  7 

Mixed  Graph  Support 

3 

Question  3c 

Probability 

1 

Equations 

2 

Question  4  (b,c,d) 

Learning 

3 

Other 

3 

Noisy-Or 

1 

Information  for  this  was 
gathered  later  on 

Virtual  Evidence  Support 

2 

Information  for  this  was 
gathered  later  on 

Inference  Algorithm 

2 

Exact 

1 

Question  6 

Approximate 

2 

Structure  Learning 

3 

Question  2b 

Figure  5.  Normalized  relative  weights  for  Input  Manipulation  criteria 
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Category.  -  Performance  and.  Extensibility 


Table  5.  Performance  and  Extensibility  Criteria  and  Relative  Weights 


Criteria 

Rel  Wt 

Subcriteria 

Rel  Wt 

Questionnaire 
questions  used 
from  Performance  & 
Extensibility 
category 

API  Availability 

3 

Question  3 

GUI 

2 

C/C++ 

1 

Question  la 

Java 

2 

Computational  Engine 

1 

C/C++ 

1 

Question  1b 

Java 

2 

J0.125  API  Availability  |- 

[1.000  Perform&Extensibilty 

^|0.125 JavaGui  | 

N 

s.  - 10.375  GUI  H 

- [0.250  O+Gui  | 

X 

^H0.500  CompEnqine 

- - 10.167  Java  | 

.333  C/C++  | 

Figure  6.  Normalized  relative  weights  for  Performance  &  Extensibility 

criteria 
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Category  -  Interoperability 


Table  6.  Interoperability  Criteria  and  Relative  Weights 


Criteria 

Rel 

Wt 

Subcriteria 

Rel 

Wt 

Questionnaire  questions 
used  from  Interoperability 
category 

Database  Input  Access 

2 

Use  ODBC 

1 

Question  1(a,b,c) 

Direct 

3 

Save  Models 

2 

SQL  Interface 

3 

Export  Analysis  Report 

1 

Question  4 

OpenModelFormat 

2 

Question  2 

Export/lmport  Model 

3 

Question  3.  Modified 
question  by  grouping  the 
ability  to  Import  nets  built  by 
other  applications. 

Figure  7.  Normalized  relative  weights  for  Interoperability  criteria 
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4.5.3  Final  goal  -  Selecting  the  optimal  Bayesian  tool 

Table  7.  Summary  Criteria  and  Relative  Weights 


Criteria 

Rel  Wt 

General  Features 

3 

Input  Manipulation 

1 

Performance  & 
Extensibility 

2 

Interoperability 

2 

Figure  8.  Normalized  relative  weights  for  summary  criteria 


4.5.4  Ranking  the  Bayesian  Tools: 

The  tools  were  ranked  against  all  the  criteria/subcriteria  in  the  four  decision 
models.  Ranks  were  assigned  based  on  the  information  gathered  about  the  tool 
and  the  extent  to  which  a  tool  could  perform  the  criteria.  For  example,  with 
criteria  like  Maturity  of  tool  greater  than  five  years  (in  the  GeneralFeatures  sub¬ 
goal  model),  tools  were  simply  assigned  100%  if  they  were  in  the  market  for 
greater  than  five  years  and  assigned  0%  for  the  subcriteria  Maturity,  less  than 
five  years.  But  for  criteria  like  ExportAnalysisReports  (in  the  Interoperability  sub¬ 
goal  model),  although  all  the  tools  provided  some  capability  to  export  analysis 
reports,  there  was  a  wide  range  in  the  extent  to  which  it  could  be  done.  For 
criteria  like  these,  we  assigned  scores  based  on  the  extent  of  capabilities 
provided  by  the  tool  and  not  just  based  on  if  the  tool  said  yes  to  the  criteria  in  the 
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questionnaire.  The  following  tables  (Tables  8-11)  summarize  the  ranks  assigned 
to  the  tools.  Each  table  is  followed  by  additional  notes  that  explain  in  detail  some 
of  the  scores  assigned. 

Category  -  General  Features 


Table  8.  Tool  Rank  for  General  Features  Criteria 


Tool  Name 

Maturity 

OS 

Support 

Architecture 

>  5  yrs 

<  5  yrs 

MultipleOS 

Client-Server 

Analytica 

100 

0 

75 

75 

BayesBuilder 

0 

100 

0 

75 

BayesiaLab 

0 

75 

100 

0 

Dxpress 

100 

0 

0 

100 

Ergo 

100 

~0~ 

100 

0 

Hugin 

100 

0 

100 

75 

Netica 

100 

0 

100 

75 

SIAM 

100 

0 

75 

100 

Bnet2000 

50 

0 

100 

0 

Additional  Notes: 

-  For  subcriteria  <  5  yrs, 

-  BayesiaLab  is  given  a  score  of  75  because  we  wanted  to  differentiate 
between  tools  that  are  about  4-5  years  old  versus  tools  that  have  1-2 
years  of  market  maturity. 

-  Bnet2000  is  given  a  score  of  50  because  this  tool  was  only  used  within 
Charles  River  Analytics  (CRA)  Inc,  and  does  not  have  any  commercial 
maturity  outside  CRA. 

-  For  subcriteria  Client-Server  support,  Analytica,  BayesBuilder,  Hugin  and 
Netica  are  given  a  score  of  75  because  there  is  no  built  in  feature  for  Client- 
Server  Architecture.  This  is  accomplished  using  an  Application  Program 
Interfaces  (API). 

-  For  subcriteria  MultipleOS,  we  were  distinguishing  between  tools  that 
supported  different  kinds  of  OS  (Windows,  Linux,  Mac)  versus  that  supported 
only  one  kind  of  OS.  A  score  of  100  was  given  if  the  tool  supported  Windows, 
Linux  and/or  Mac. 

-  Although  Analytica  supports  more  than  one  OS  (Windows  and  Mac),  it 
was  given  a  score  of  75  because  it  did  not  support  Linux  (Customer 
emphasized  that  the  tool  has  support  for  Linux). 

-  Although  SIAM  is  meant  to  support  Windows  and  Linux  (since  it’s 
developed  in  Java),  its  not  tested  on  Linux.  Due  to  this,  it  was  given  a 
score  of  75. 
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Category  -  Performance  and  Extensibility : 


Table  10.  Tool  Rank  for  Performance  and  Extensibility  Criteria 


Computational 

Engine 

GUI 

API 

Availability 

C/C++ 

Java 

C++ 

Java 

Analytica 

100 

0 

100 

0 

100 

BayesBuilder 

100 

0 

0 

100 

100 

BayesiaLab 

0 

100 

0 

100 

100 

Dxpress 

100 

0 

100 

0 

100 

Ergo 

100 

0 

100 

0 

100 

Hugin 

100 

0 

0 

100 

100 

Netica 

100 

0 

100 

0 

100 

SIAM 

\o~ 

100 

0 

100 

0 

Bnet2000 

0 

100 

0 

100 

75 

Additional  Notes: 

-  For  the  tool  performance  availability  criteria,  scores  were  assigned  based  on 
the  information  that  was  made  available  to  us  by  the  tool  vendor. 

-  For  criteria  API  availability,  Bnet2000  was  given  a  score  of  75  because 
negotiation  is  involved. 

Category.  -  Interoperability 


Table  11.  Tool  Rank  for  Interoperability  Criteria 


Tool  Name 

Databases 

Export 

Analysis 

Reports 

OpenModel 

Format 

Export/ 

Import 

Models 

Direct 

ODBC 

Save 

Models 

SQL 

Interface 

Analytica 

0 

100 

100 

100 

100 

100 

100 

BayesBuilder 

0 

0 

0 

0 

25 

0 

100 

BayesiaLab 

0 

100 

0 

100 

100 

100 

100 

Dxpress 

0 

0 

100 

0 

25 

0 

0 

Ergo 

0 

0 

0 

0 

75 

75 

0 

Hugin 

100 

100 

0 

0 

50 

75 

100 

Netica 

100 

100 

0 

100 

75 

75 

100 

SIAM 

IP 

0 

0 

0 

100 

100 

100 

Bnet2000 

0 

0 

0 

0 

75 

100 

100 

Additional  Notes: 

-  For  the  criteria  ExportAnalyeieReporte  scores  were  assigned  based  on  the 
following: 

-  If  the  tool  provided  a  graphical  display  of  what-if  and  inference  results; 
capability  to  copy  and  paste  the  network  to  other  applications  (MS 
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Word,  PowerPoint,  etc)  by  a  simple  click  and  drag,  it  was  a  given  a 
score  of  100. 

-  If  the  tool  primarily  generated  textual  reports  and  allowed  the  user  to 
select  and  paste  the  results  and  the  nodes  from  other  networks,  to 
other  application  (click  and  drag),  it  was  given  a  score  of  75. 

-  If  the  tool  allows  the  user  to  save  the  network  and  the  results  as  a  file 
that  could  be  opened  later,  it  was  given  a  score  of  50. 

-  If  the  tool  allowed  the  user  to  only  save  the  case  files,  it  was  given  a 
score  of  25. 

-  For  the  criteria  Open  Model  Format,  Ergo  is  given  a  score  of  75  because 
the  open  format  feature  is  not  as  efficient  as  saving  the  model  in  the 
proprietary  format;  Although  Hugin  does  not  support  any  standard  formats 
(XML,  XBN,  BIF  etc),  it  is  given  a  score  of  75  because  the  format  it 
supports,  .net,  is  exportable  to  many  other  Bayesian  Net  tools;  Netica  is 
given  a  score  of  75  because  the  Open  Model  Format  feature  is 
implemented,  but  not  yet  made  available  to  the  users. 


4.5.5  Discussion  of  the  Results: 

Results  were  generated  in  two  phases.  The  first  phase  involved  generating 
scores  for  the  tools  within  the  four  sub-goal  models  -  General  Features,  Input 
Manipulation,  Performance  &  Extensibility  and  Interoperability.  These  results 
were  then  combined  together  into  a  final  decision  model,  as  seen  in  Figure  8. 
The  four  criteria  in  Figure  8  are  essentially  soft-links  to  the  four  sub-goal  models. 
The  results  from  the  sub-goal  models  are  propagated  to  the  final  model,  where 
they  are  combined  to  generate  final  decision  scores. 

For  every  decision  model,  CDP  displays  the  results  in  a  decision  score 
window.  A  decision  score  window  displays  the  decision  score  of  each  of  the 
alternatives  in  the  model  in  a  horizontal  bar  chart.  Each  line  shows  the  name  of 
the  alternative,  the  value  of  the  decision  score,  and  a  horizontal  bar  reflecting  the 
value  of  the  decision  score  [1]. 

For  further  analysis  of  the  decision  scores,  CDP  provides  a  feature  that 
displays  a  breakdown  view  of  contributions  from  each  of  the  criteria  toward  the 
decision  score  of  the  alternative.  Following  are  decision  score  windows  and 
contribution  by  criteria  windows  (Figures  9-16)  for  all  the  sub-goal  models  and 
the  final  model. 
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Category  -  General  Features  [GF} 


Decision  GeneiaFeatues 


Figure  9.  GF  Decision  Score 


|  Architecture 
|  Maturity 
|  OS  Support 


Figure  10.  GF  Contribution  by  Criteria 
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Category  -  Input  Manipulation  (IM) 


Doctxon  InputManiputabon 


Figure  11.  IM  Decision  Score 


0.25 


0.25 


B  StructLearning 
|  MixedGraphSupport 
|  VirtualEvidence 
O  Probabilities 
B  AnalysisTech 
U  DecisionNodeSupport 
|  InferenceAlgorithm 


Figure  12.  IM  Contribution  by  Criteria 
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Category  -  Performance  Extensibility  [PE} 


Figure  13.  PE  Decision  Score 
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Category  -  Interoperability  [10} 


— —  Decisionr  Intaoperabity - 


Figure  15.  10  Decision  Score 


|  Databases 
|  ExpAnalysisReports 
|  OpenModelFormat 
I  |  ExportlmportModels 


Figure  16.  10  Contribution  by  Criteria 
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4.5.6  Scores  from  integrating  the  four  sub-models  into  a  final  model 


Figure  17.  Integrated  Decision  Score 


0.16 

0.14 

0.12 

0.10 

0.08 

0.06 

0.04 

0.02 

0.00 


|  Link  -  InputManipulation 
|  Link  -  Interoperability 
|  Link  -  Perform&Extensibil 
I  I  Link  -  GeneralFeatures 


Figure  18.  Integrated  Contribution  by  Criteria 

From  the  final  goal  decision  score  window  (Figure  17),  Hugin  ranks  as  the 
optimal  Bayesian  tool  among  all  the  alternatives  (though  others  are  close).  From 
the  final  goal  contribution  by  criteria  window  (Figure  18),  it  can  be  further  seen 
that  the  score  from  Input  Manipulation  criteria  had  the  maximum  contribution 
towards  the  decision  score  of  Hugin.  The  reason  for  this  is  that  the  criteria  Input 
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Manipulation  carried  the  maximum  weight  with  respect  to  the  final  goal  of 
choosing  an  optimal  tool  and  within  the  Input  Manipulation  decision  model,  it  can 
be  seen  that  Hugin  ranked  the  highest.  This  gave  Hugin  a  lead  compared  to  the 
other  tools. 

BayesiaLab  and  Netica  are  the  next  best  performance  alternatives.  Once 
again,  the  reader  can  look  at  the  contributions  by  criteria  window  to  gain  further 
insight  into  the  contributing  factors  towards  the  final  decision  scores  of  these 
tools. 

4.5.7  Discussion  of  GOTS  Tools 

4.5.7. 1  SAIC 

SIAM 

Compared  to  the  other  commercial  Bayesian  tools,  SIAM  ranked  5th  among 
the  nine  alternatives.  Following  are  the  reasons  for  its  relatively  low  decision 
score: 


•  Input  Manipulation:  One  of  the  major  drawbacks  of  SIAM  is  that  it  does  not 
support  Bayesian  Inferencing.  Bayesian  Inferencing  is  where  the  user, 
upon  collection  of  more  information  about  certain  nodes  in  the  net,  enters 
the  information  (evidence)  into  the  net.  Based  on  the  new  information 
entered,  the  tool  updates  the  probabilities  of  other  nodes  in  the  network. 
Some  of  the  other  features  that  SIAM  does  not  address  the  learning  of 
probabilities  from  a  sample  set  and  learning  of  the  structure  of  the  net 
from  a  database.  These  features  are  useful  to  have  if  the  end  user  wants 
the  tool  to  work  with  the  existing  data. 

•  Performance  &  Extensibility:  In  this  category,  SIAM  ranked  low  because  of 
the  emphasis  placed  on  the  language  used  to  develop  the  computational 
engine  and  the  Graphical  User  Interface  (GUI).  The  preferred  language 
was  C/C++  and  SIAM  was  developed  in  Java.  The  lack  of  an  Application 
Program  Interface  (API)  for  SIAM  also  contributed  to  its  low  score. 

4. 5. 7.2  Charles  River  Analytics 

Organizational  &  Cultural  Criteria  for  Adversary  Modeling  -  Bnet2000 

OCCAM  is  a  decision  aiding  system  that  helps  build  models  by  employing 
Bayesian  Network  Techniques,  Rule  Based  Reasoning  and  Social  Net  Analysis. 

The  following  steps  outline  how  these  techniques  work  together  in  building  a 
decision  aiding  system.  (Note:  This  high  level  information  is  gathered  from  talking 
to  the  developers  of  OCCAM  and  not  from  using  the  tool  or  reading  any  technical 
description  of  the  tool): 
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•  An  Entity  is  a  person  or  organization  that  the  user  is  interested  in 
modeling.  An  Entity  has  attributes. 

•  Application  data  is  collected  for  the  attributes  of  the  entity. 

•  Rules  are  constructed  using  this  application  data  and  also  the  data  is 
further  analyzed  using  Social  Net  Analysis  tools. 

•  The  next  step  is  to  build  a  Bayesian  net  using  Bnet2000.  Bnet2000  is  an 
in-house  Bayesian  tool.  Currently,  Bnet2000  does  not  support  automatic 
construction  of  the  Bayesian  net  from  the  data  available.  So  the  net  is  built 
bv  domain  experts  who  understand  the  causal  relationships  between  the 
nodes.  Beliefs  for  the  nodes  are  recorded  based  on  the  results  from  rule 
based  reasoning,  social  net  analysis  of  the  application  data  and  other 
proprietary  techniques. 

As  OCCAM  employs  more  than  Bayesian  Nets,  we  felt  that  it  needs  to  be 
addressed  separately  and  not  be  included  in  the  list  of  alternatives  being  rated. 
However,  we  did  include  Bnet2000  to  the  list  of  alternatives.  From  the  decision 
scores  it  can  be  seen  that  it  did  not  rank  high.  The  reason  for  low  scores  is  partly 
due  to  the  fact  that  we  did  not  have  an  evaluation/trial  copy  to  work  with  and  had 
to  entirely  depend  on  the  information  from  the  questionnaire.  Also,  it  might  be 
that  Bnet2000  did  not  address  some  features  due  to  the  fact  that  it  is  being  used 
together  with  other  Al  techniques,  where  the  capabilities  from  these  techniques 
compensated  for  the  features  that  Bnet2000  lacks. 

Based  on  the  information  gathered  for  OCCAM,  it  can  be  said  that  it  seems 
to  be  addressing  the  domain  specific  needs  of  the  customer.  It  also  employs  all 
the  techniques  that  the  customer  is  planning  to  apply  to  build  a  decision  support 
system  that  helps  model  the  influences  impacting  human  behavior. 

Sample/Grade 

Sample  is  an  architecture  that  provides  support  to  the  user  to  build  models 
that  apply  Artificial  Intelligence  (Al)  techniques.  Some  of  the  techniques 
supported  are  Bayesian  Nets,  Fuzzy  Logic,  Genetic  Algorithms,  and  Expert 
Systems  etc.  Grade  is  a  GUI  development  environment  that  helps  the  user  build 
a  model  by  combining  the  techniques  provided  by  Sample  and  to  later  simulate  it. 
At  SRA,  we  haven’t  had  a  chance  to  work  with  this  architecture/environment  tool. 
From  the  information  we  collected,  we  believe  that  it  might  be  useful  to  the 
customer  if  they  plan  to  employ  multiple  Al  techniques  other  than  Bayesian  Nets. 
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4. 5. 7.3  AFRL/Metrica,  Inc. 


Evaluation  of  Cross-Cultural  Models  for  Psychological  Operations:  Cross-Cultural 

PSYOP  Decision  Support  System 

This  system  was  developed  as  a  part  of  a  study  performed  by  Metrica,  Inc., 
for  Air  Force  Research  Laboratory.  The  study  was  done  in  three  phases,  where 
the  last  phase  led  to  the  development  of  the  Cross-Cultural  PSYOP  Decision 
Support  System  ( CCP-DSS ).  Refer  to  [2]  (at  end  of  Section  4)  for  the  details  of 
the  study. 

The  CCP-DSS  is  a  web-based  system  that  merges  several  databases 
related  to  PSYOP  supporting  objectives  included  in  the  system.  These  data 
include  the  associated  influencing  factors  for  the  objectives,  background 
information  about  the  individual  factors  and  response  data  from  previous  users, 
detailing  their  assessments  of  factor  influence  and  probable  target  audience 
response  to  specific  hypothetical  PSYOP  scenarios  [2]. 

This  DSS  was  not  considered  as  one  of  our  alternatives  because  of  the 
following: 

•  Employs  SMART  technique  and  not  Bayesian  methods.  We  are  primarily 
focused  on  tools  that  employ  Bayesian  techniques. 

•  It  is  a  Knowledge-driven  DSS.  These  are  systems  that  have  specialized- 
problem  expertise.  The  system  contains  knowledge  about  a  specific 
domain  and  uses  this  knowledge  to  aid  decision  process  of  the  user.  The 
alternatives  we  are  considering  are  not  Knowledge-driven  DSS. 

Although  the  tool  is  not  applicable  as  one  of  the  alternatives,  the  feature  of 
building  a  database  that  contains  data  supporting  PSYOP  objectives  and  helping 
the  user  based  on  this  information  is  a  very  useful  technique.  This  technique  can 
be  applied  to  the  Bayesian  tools  to  help  the  user  determine  the  a-priori 
probabilities. 

4.5.8  Future  Extensions 

For  the  evaluation  process  of  the  tools,  we  did  not  consider  features  of  the 
tools  that  were  in  the  process  of  being  implemented.  However,  we  feel  that  it  is 
important  to  mention  these  extensions  so  that  the  customer  can  decide 
accordingly  -  to  go  with  the  current  optimal  tool  or  to  wait  for  a  tool  (that  is 
relatively  cheap)  and  will  have  certain  features  implemented  in  their  next 
versions.  In  general,  with  the  commercial  tools,  tool  vendors  were  very  flexible 
regarding  extension  of  their  feature  set  to  accommodate  customer  needs.  Table 
12  summarizes  the  tools  and  future  extensions  in  their  next  releases.  This 
information  was  gathered  from  talking  to  the  respective  tool  representatives. 
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Table  12.  Planned  Extensions  for  Some  Vendors 


Tool  Name 

Extensions 

Netica 

Availability  of 
OpenModel  Format 
and  Structure 

Learning 

BayesiaLab 

Availability  of 

Decision  Nodes  and 
Conflict  Analysis 

Ergo 

Sensitivity  Analysis 
can  be  performed  via 
GUI 

4.5.9  “Ease  of  Use”  Evaluation 
4. 5.9.1  Industry  Standard  Techniques 

The  usability  evaluation  was  conducted  on  the  four  demo  version 
applications  that  ranked  highest  in  the  functional  evaluation:  Analytics,  Netica, 
Hugin,  and  BayesiaLab.  Three  primary  approaches  to  usability  assessment  were 
considered  in  determining  an  evaluative  methodology  for  this  project:  1 )  heuristic 
evaluations  with  usability  guidelines,  2)  cognitive  walkthroughs,  and  3)  usability 
testing.  The  selected  methodology  included  elements  of  all  three  techniques. 
Testing  was  conducted  by  a  human  factors  researcher;  the  results  were 
collapsed  and  loaded  into  the  decision  support  tool  to  obtain  a  usability  ranking. 

A  short  discussion  of  the  three  methods,  followed  by  a  discussion  of  the  selected 
evaluation  methodology,  the  ranking  criteria,  and  the  evaluation  results  follow. 

Heuristic  evaluations  employ  general  usability  guidelines  (testable  design 
principles)  that  focus  attention  on  design  areas  that  have  historically  proven 
sources  of  user  difficulties.  A  good  heuristic  evaluation  attempts  to  balance 
consistency  (maintaining  standardization  for  ease  of  use)  and  complexity 
(allowing  variation  to  support  expert  users)  and  assess  the  application  in  the 
context  of  its  expected  use.  Table  13  lists  ten  commonly  accepted  usability 
guidelines,  derived  by  factor  analysis  of  249  usability  problems  across  1 1 
projects  [6].  Cost/benefit  analyses  have  shown  that  the  most  effective  results  are 
obtained  using  four  evaluators  for  a  comprehensive  result  [7].  Problems  with  this 
method  involve  assessing  how  to  deal  with  trade-offs  and  how  to  apply 
guidelines  judiciously  within  the  current  context. 
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Table  13.  Ten  Commonly  Accepted  Usability  Guidelines 


Visibility  of  system  status.  System  provides 
status  feedback  to  user. 

Recoqnition  rather  than  recall.  System  keeDS  all 
options  and  support  for  actions  visible. 

Match  between  system  &  the  real  world.  System 
employs  familiar  terms  and  concepts. 

Help  recoanizina,  diaanosinq  &  recoverinq  from 

errors.  System  assists  rather  than  confuses. 

User  control  &  freedom.  System  offers  auick  exit 
path  from  errors  and  undo/redo  function. 

Flexibility  &  efficiency  of  use.  System  offers 
accelerators  for  expert  users  while  aiding  novices. 

Consistency  &  standards.  System  uses 
consistent  terms  and  visualizations. 

Aesthetic  &  minimalist  desion.  System 
communications  are  relevant  and  cogent. 

Error  prevention.  System  desian  helps  user 
avoid  errors. 

Help  &  documentation.  System  helps  are  concise, 
clearly  stated,  task-focused,  and  searchable. 

Cognitive  walkthroughs  employ  a  set  of  representative  tasks;  the  evaluator 
performs  the  tasks,  step-by-step,  and  documents  each  system  interaction.  In  a 
formal  cognitive  walkthrough,  the  evaluator  uses  pre-prepared  forms  to 
document  the  typical  user’s  specific  goals,  tasks,  and  knowledge  at  each  step. 
How  the  system  interface  appears  and  how  it  changes  in  response  to  user 
actions  is  also  documented.  This  form  of  evaluation  is  based  on  the  psychology 
of  inexperienced  users  and  evaluates  the  system  for  ease  of  learning  and 
support  for  exploratory  learning.  Identification  of  the  psychological  bases  for  user 
difficulties  (e.g.,  confusion  due  to  inadequate  function  labeling  or  error  due  to 
insufficiently  explained  input  options)  also  indicates  possible  solutions.  Testing  is 
based  on  four  main  questions:  1 )  Will  the  users  try  to  achieve  the  right  effect? 

2)  Will  the  users  notice  the  correct  action  is  available?  3)  Will  the  user  associate 
the  correct  action  with  the  desired  effect?  and  4)  If  the  user  performs  correctly, 
will  the  user  realize  progress  is  being  made  toward  task  accomplishment?  The 
problems  with  cognitive  walkthroughs  center  around  three  points.  They  require 
in-depth  knowledge  of  the  task  domain  for  adequate  test  task  selection.  They  are 
time-consuming;  strict  protocols  and  copious  documentation  require  time.  They 
are  also  susceptible  to  a  low-level  problem  bias;  concentrating  at  the  keystroke 
level,  it  is  difficult  to  see  the  high  level  problems  that  yield  higher  payoffs. 

Usability  testing  involves  the  development  of  empirical  test  plans  that  allow 
observers  to  assess  system  support  to  real  users  in  realistic  scenarios, 
performing  real-world  tasks.  Observers  document  user  behaviors  and  evaluate 
performance  effectiveness;  they  also  may  employ  a  “thinking  aloud”  protocol  to 
access  user  thought  processes.  Interviews  and  questionnaires  are  also 
techniques  used  to  capture  user  impressions.  Studies  support  three  users  as  a 
cost  effective  number  of  test  subjects  [7].  Problems  with  this  method  involve  the 
development  of  appropriate  scenarios  and  tasks,  difficulties  with  subject 
availability,  and  interpretation  of  results  from  different  user  expertise  levels. 
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4. 5.9.2  Integrated  Approach 

This  project  did  not  support  the  full-scale  employment  of  any  of  the  foregoing 
techniques.  Neither  multiple  domain  experts  nor  multiple  usability  experts  were 
available;  time  considerations  were  also  a  limiting  factor,  constraining  both  test 
duration  and  model  task  development.  To  meet  project  needs,  a  rapidly 
obtainable,  high-level  usability  evaluation  was  created,  drawing  on  elements  in  all 
three  techniques.  The  test  plan  created  for  this  project  was  based  jointly  on  two 
industry  standard  heuristics,  Xerox  Corporation’s  Usability  Analysis  and  Design, 
Heuristic  Evaluation:  A  System  Checklist  (an  expansion  of  the  ten  factors 
identified  earlier)  [8]  and  DoD’s  Defense  Information  Infrastructure  Common 
Operating  Environment  (DIICOE)  User  Interface  System  Checklist  [9].  The  two 
checklists  were  examined  for  overlaps  and  edited  accordingly.  A  preliminary 
examination  of  the  applications  indicated  that  many  of  the  low-level  detailed 
checks  were  either  not  problematic  or  subordinate  in  importance  to  larger  issues; 
the  combined  checklist  was  filtered  for  non-issues  and  subordinate  issues  to 
leave  the  items  likely  to  represent  critical  design  concerns.  The  resulting  list 
incorporated  “bellwether”  issues  from  all  of  the  major  categories  (Table  14).  The 
matrix  in  Attachment  9  offers  more  detail. 


Table  14.  List  of  Significant  Evaluative  Issues 


No.  Issue 

4b  Error  Prevention/Recovery 

4c  System  Shutdown 

5  Flexibility/Efficiency/Legibility 

6  Other  Functionality 

7  Visualizations 

8  Print  Control  &  Reports 

9  Help/Documentation 

9a  Type  of  Online  Help  file 

9b  Information  Organization 

9c  Accessibility  of  Language 

9d  Tutorial 


No.  Issue 

1  User/Computer  Interaction 
la  Icons/Push  Buttons 

1b  Menus 

1c  Windows/Oialog  Boxes 

Id  Feedback 

2  Consistency/Standards 

2a  Windows  Metaphor 

2b  Common  Look  &  Feel 

3  System  Status  Visibility 

4  Errors,  Error  Prevention  &  Recovery 

4a  Errors 


Although  grouped  somewhat  differently  to  better  correspond  with  potential 
problem  areas  in  the  applications,  all  but  one  of  the  ten  factors  were  explicitly 
reviewed.  “Match  between  system  and  real  world”  was  not  considered  truly 
appropriate  to  the  current  evaluation.  The  terminology  for  Bayesian  statistics  and 
Bayesian  net-building  was  domain-specific;  in  this  case  it  seemed  more  realistic 
to  check  documentation  for  clarity  of  language  and  concept  explanations.  The 
lead  issues  (each  with  more  detailed  subcriteria)  were  incorporated  in  a  matrix; 
each  application  was  reviewed  separately  and  results  and  comments  were  noted. 

The  selected  approach  integrated  elements  from  the  other  usability 
evaluation  techniques  as  well.  The  concepts  of  test  tasks  and  bi-level  analysis 
(low  level  data  entry  as  well  as  more  high-level  functional  organization  and  task 
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support  issues)  were  drawn  from  cognitive  walkthroughs  and  the  documentation 
structure  of  the  effort,  from  observational  documentation  methods.  In  the 
absence  of  a  formal  test  model,  three  methods  were  used.  First,  in  keeping  with 
the  psychological  orientation  of  the  heuristic  evaluation,  the  evaluator  tried  to 
exercise  baseline  network  construction  and  compilation  functions  as  a  novice, 
looking  for  identifiable  tools  and  intuitive  processes.  Difficulties  were  noted  and 
assessed.  Second,  the  evaluator  attempted  to  build  a  small  network  and  compile 
it.  Third,  the  evaluator  opened  the  Asia  model  (a  diagnostic  model  included  with 
each  application  as  an  example)  and  systematically  altered  it  to  observe  the 
system  response.  All  results  were  integrated  in  the  relevant  sections  of  the 
evaluation  matrix. 

The  initial  documentation  effort  was  not  set  up  to  be  hierarchical.  However, 
after  reviewing  the  results,  it  was  determined  that  MCDA  AHP  could  be  applied  to 
the  usability  criteria  as  well.  Accordingly,  the  documentation  was  examined  for 
trends  and  for  uniform  scores.  Where  all  applications  were  rated  uniformly 
acceptable,  the  criterion  was  dropped  from  further  consideration.  Where  all 
applications  were  rated  uniformly  unacceptable,  the  results  were  analyzed  and 
the  criterion  was  expanded  appropriately  to  try  to  capture  the  cause  of  the 
problem.  Trends  were  evaluated  for  contributory  factors  and  the  criteria  list  was 
amended  to  probe  those  issues.  The  amended  list  was  decomposed  so  each 
criterion  could  be  weighted  appropriately.  As  all  the  factors  in  the  final  criteria  list 
were  considered  critical  to  good  user  interface  design,  weights  were  distributed 
relatively  equally  across  the  criteria.  The  final  evaluative  criteria  were 
reassessed  for  each  program  and  any  changes  in  results  were  integrated  into 
both  the  initial  and  final  criteria  sets. 
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Table  15.  Final  Evaluative  Criteria  List 


1.  User  Support 

Dialog  Box  Help 
Find  Function 
Wizards 
T  utorial 

Example  Scenarios 
Level  of  Complexity 
Process  Orientation 
Language 

Key  Word  Search  Capability 

Help 

Language 
Glossary 
Searchable  Index 
TOC  Organization 
Completeness 


2.  Error  Management 
Validation  Errors 

Warnings 

Explanations 

Automatic  Repair  Function 

Input  Errors 
Warnings 
Explanations 
System  &  Fatal  Errors 

3.  Efficiency  Factors 

Rapid  Input/Manipulation  Methods 
Repetitive  Action  Capabilities 
Align  Tool 
Reverse  Links  Tool 
Windows  Common  Look/Feel 
Comprehensive  Node  Representation 
Model  Description 


4.  Visualizations 
On  Screen 
Tables  &  Networks 
Graphs 
Text 

Error  Logs 

Printing 

Page-break  Preview 
Scale-to-Fit  Capability 
Completeness 


The  final  criteria  list  (Table  15  above)  excluded  most  Windows 
Metaphor/Common  Look  and  Feel  issues.  There  were  few  problems  with  the 
Windows  metaphor  or  standardization  of  application  elements:  it  was  collapsed 
to  a  single  entry  under  Efficiency  Factors.  Feedback  was  not  an  issue  either 
because  it  was  uniformly  adequate  or  uniformly  untestable  with  the  available 
models.  Icons  were  not  a  significant  factor  in  user  problems.  Major  trends  were 
in  application  documentation  and  internal  support  to  task  accomplishment — both 
judged  critical  application  usability.  Error  messages  offered  some  problems: 
system  design  was  insufficient  to  eliminate  errors,  but  without  a  more  detailed 
model  it  was  difficult  to  probe  further.  Print  control  and  print  capabilities  were 
insufficiently  supported;  the  total  number  of  visualizations  each  application 
offered  were  difficult  to  document  and  display.  Efficiency  factors  were  less 
critical  to  task  completion,  but  certain  features  (e.g.,  reverse  links  and  rapid 
netbuilHing  took)  wnnlrl  ho  very  important  to  large  network  construction.  The 
weighted  evaluation  criteria  and  AHP  results  are  discussed  in  the  sections  below. 
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4. 5. 9. 3  Ease  of  Use  Results  (Tables  16-27) 

User  Support  Evaluation  Criteria 


Table  16.  Definitions  for  User  Support  Criteria 


User  Support 

Dialog  Box  Help 
Find  Function 
Wizards 
Tutorial 

Example  Scenarios 
Level  of  Complexity 

Process  Orientation 
Language 

Key  Word  Search  Capability 

Help 

Language 

Glossary 
Searchable  Index 

TOC  Organization 

Completeness 


Description 

Offers  interpretative  help  filling  in  data  fields  from  within  the  dialog  box. 
Permits  modeler  to  search  for  a  specific  node  in  a  large  network. 
Facilitates  performing  routine  or  specialized  action  sequences. 

Tutorial  provides  example  scenarios  for  all  supported  utilities. 
Scenarios  have  sufficient  complexity  to  exercise  all  functions. 

Tutorial  language  displays  a  process  rather  than  a  procedural 
orientation  (step-by-step  instructions  rather  than  overview). 

Language  is  simple,  direct,  and  uses  as  little  jargon  as  possible. 
Tutorial  has  internal  keyword  search  capability  to  support  finding 
specific  instructions. 

Language  is  simple,  direct,  and  uses  as  little  jargon  as  possible. 

Help  includes  a  glossary  of  terms  to  facilitate  understanding  of 
technical  terminology. 

Help  includes  a  searchable  index  of  all  key  terms. 

Table  of  Contents  is  organized  to  support  reasonable  sequence  of 
modeler  activities  (either  working  with  or  creating  networks). 

All  functions  and  required  inputs  are  completely  explained  in  Help. 
Tutorials  offer  supplementary  method  of  obtaining  information. 


Table  17.  Rank  Structure  for  User  Support  Criteria 


User  Support 


Dialog  Box  Help 
Find  Function 

Wizards 

Tutorial 

Example  Scenarios 

Level  of  Complexity 

Process  Oi  iei ilcaliui i 

Language 


Rank  Definition 

0=  No  dialog  box  Help  and  can't  open  Main  Help  w/dialog  box  open. 
25=No  dialog  box  Help  but  can  open  Main  Help  w/dialog  box  open. 
50=Dialog  box  Help  links  to  Main  Help  table  of  contents. 

75=Dialog  box  Help  links  to  appropriate  topic  in  Main  Help. 
100=Dialog  box  Help  opens  popup  explanations/instructions. 

Y=  100 
N=0 
Y=  100 
N=0 


75  =  Most 
100=  All 

Note:  Can't  tell  if  all  functions  are  included  in  this  level  of  audit. 
Basic  functions=50 
Complex  functions=75 
All  functions=100 

Note:  Can’t  tell  if  exercises  all  functions  with  this  level  of  audit. 
Y=  100 

N-0 

Clear  Language=100 
Somewhat  Clear=50 
Not  Clear=0 
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User  Support 

Key  Word  Search  Capability 

Help 

Language 

Glossary 

Searchable  Index 
TOC  Organization 

Completeness 


Rank  Definition 
Whole  documents  00 
By  Chapter=75 
By  Page=25 

Clear  Language  =  100 

Somewhat  Clear=50 

Not  Clear=0 

Y=  100 

N=0 

Y=100 

Some  search  capability=50 

N=0 

Y=  100 

N=0 

Completed  00 
Reasonably  Complete=75 
Somewhat  Complete=50 
Not  Complete=25 


User  Support  Resujts 


Table  18.  Results  for  User  Support  Criteria 


User  Support 

Analytica 

Netica 

Hugin 

BayesiaLab 

Dialog  Box  Help 

25 

50 

25 

0 

Find  Function 

100 

100 

0 

100 

Wizards 

0 

0 

100 

100 

Tutorial 

Example  Scenarios 

100 

50 

100 

50 

Level  of  Complexity 

100 

50 

100 

50 

Process  Orientation 

100 

0 

100 

100 

Language 

100 

100 

50 

100 

Key  Word  Search  Capability 

100 

75 

75 

100 

Help 

Language 

100 

100 

50 

0 

Glossary 

100 

0 

0 

0 

Searchable  Index 

100 

100 

0 

50 

TOC  Organization 

100 

100 

0 

0 

Completeness 

75 

50 

25 

50 

All  applications  were  distinctly  lacking  in  context-sensitive  Help,  a  very 
important  user  support  function.  An  ideal  software  application  provides 
substantive  assistance  with  all  decision  points;  this  includes  directions  for  how  to 
fill  in  all  data  fields.  The  criterion  Recall  vs.  Recognition,  one  of  the  ten  standard 
usability  guidelines,  stresses  the  importance  of  keeping  support  information 
visible;  this  is  an  important  error  avoidance  technique.  In  the  four  applications 
evaluated  the  major  need  for  context-sensitive  Help  was  in  dialog  box 
interactions;  therefore,  the  evaluation  refers  specifically  to  Dialog  Box  Help. 
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Dialog  boxes  require  the  user  to  fill  in  data  fields  to  accomplish  a  task.  While  it 
would  be  impractical  to  keep  all  data  field  descriptions  and  input 
criteria/constraints  visible,  it  is  very  possible  to  use  hover  popups  or  rightclick 
responses  to  provide  the  necessary  support.  The  worst  case  scenario  was 
shown  in  BayesiaLab,  where  not  only  is  there  no  internal  help,  but  the  main  Help 
function  cannot  even  be  accessed  without  closing  the  dialog  box  (a  clear  case  of 
recall  dependence). 

Three  of  the  four  applications  included  a  Find  function,  permitting  users  to 
search  for  and  locate  specific  nodes  by  name;  no  capability  was  identified  in 
Hugin.  This  feature  could  become  important  in  large,  complex  network 
constructions.  In  contrast,  only  Hugin  and  BayesiaLab  offered  any  kind  of 
Wizards — Hugin,  for  setting  up  structure  and  batch  learning  and  BayesiaLab,  for 
file  importation.  Wizards  are  interactive  help  utilities  that  guide  users  through 
potentially  complex  tasks,  allowing  novice  users  to  perform  at  higher  levels  of 
expertise.  Often  implemented  as  a  sequence  of  dialog  boxes  that  prompt  users 
to  fill  in  required  details,  properly  documented  wizards  are  an  excellent  way  to 
support  multiple  levels  of  user  expertise. 

The  Help  and  Tutorial  tools  are  critical  to  usability.  Help  documentation 
ideally  is  designed  around  its  potential  use,  either  as  an  on-line  support  or  as  a 
print  document;  readability  on-line  is  not  the  same  as  readability  in  print.  On-line 
Help  is  very  important  because  it  answers  user  questions  during  task 
accomplishment.  Printable  manuals  are  very  important  because  they  allow  the 
user  to  study  the  application  (and  by  extension,  task  performance)  for  extended 
periods  without  access  to  a  computer.  Tutorials  (which  also  can  be  on-line  or 
printed)  walk  users  through  example  tasks,  increasing  their  expertise. 

Information  in  tutorials  does  not  substitute  for  Help.  When  users  want  answers, 
they  want  them  immediately,  and  they  often  cannot  afford  to  search  through 
linear  tutorial  presentations  to  obtain  critical  information.  Therefore,  every  user 
action  should  be  supported  by  the  main  Help.  Tutorials  should  provide  step-by- 
step  instructions  for  both  simple,  novice-level  tasks  and  complex,  expert-level 
activities.  Both  the  main  Help  and  the  tutorial  should  be  organized  around  task 
performance,  beginning  with  frequently  performed  actions,  continuing 
progressively  through  less  frequent  activities.  The  language  in  all  documentation 
should  be  clear,  concise,  and  understandable  by  all  levels  of  user.  Domain- 
specific  terms  should  be  explained  in  context  and  in  a  glossary.  On-line  Help 
and  tutorials  should  support  key  word  searches. 

BayesiaLab  suffered  from  very  poor  translation  in  its  on-line  Help,  rendering 
it  almost  incomprehensible;  its  tutorial  was  much  more  understandable,  however. 
Hugin  used  HTML  files  for  both  its  on-line  Help  and  its  tutorial,  rendering 
searches  difficult.  It  also  used  highly  technical  language  in  places,  failed  to 

provide  a  gloccary,  offorod  no  indox  for  rapid  navigation,  and  its  table  of  contents 

was  organized  around  alphabetized  menu  and  toolbar  items  rather  than  tasks. 
Finding  task  support  through  Hugin  was  extremely  difficult.  However,  Hugin' s 
tutorial  was  very  good  in  its  provision  of  introductory  material  and  representative 
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task  instructions.  Netica’s  HTML-based  tutorial  was  also  difficult  to  search  and 
displayed  a  procedural  rather  than  a  step-by-step  process  orientation  (since  it  is 
still  under  development  it  is  unfair  to  rate  its  completeness).  None  of  the  main 
Helps  appeared  able  to  stand  alone;  task  accomplishment  required  lengthy 
searches  of  both  Help  and  tutorials;  Hugin  was  worst  as  its  on-line  Help 
appeared  to  lack  much  critical  information.  Analytica  provided  by  far  the  most 
detailed  documentation  (the  manual  is  548  pages),  including  a  glossary,  index, 
error  messages  and  a  function  list;  their  Help  and  tutorials  were  PDFs  (making 
them  both  searchable  and  printable)  with  multiple  examples  of  varying  complexity 
and  a  process  orientation  throughout.  Analytica' s  Help  provides  functional 
information  while  its  tutorial  steps  the  user  through  building  representative 
models.  The  only  weakness  encountered  was  an  occasional  need  to  use  the 
tutorial  to  supplement  Help — but  required  information  was  readily  available. 


Efficiency.  Factors  Evaluation  Criteria 

Table  19.  Definitions  for  Efficiency  Factor  Criteria 


Efficiency  Factors 

Rapid  Input/Manipulation  Methods 

Repetitive  Action  Capabilities 
Align  Tool 

Reverse  Links  Tool 

Windows  Common  Look/Feel 

Comprehensive  Node  Representation 
Model  Description 


Description 

Facilitates  rapid  net-building  by  allowing  user  to 
create  multiple  nodes  and  connectors. 

Allows  modeler  to  align  multiple  nodes. 

Allows  modeler  to  reverse  direction  of  the  links 
between  selected  nodes. 

Uses  a  Windows  metaphor  and  provides  standard 
functions  in  standardized  positions,  stable  tool 
bars,  easily  interpreted  icons,  hover  descriptions  of 
icons. 

Node  network  and  node  attributes  and  conditional 
probabilities  can  be  displayed  and  edited  on  the 
same  screen. 

Permits  modeler  to  describe  rationales  and 
assumptions  to  aid  others  using  the  model. 


Table  20.  Rank  Structure  for  Efficiency  Factors  Criteria 


Efficiency  Factors  Rank  Definition 

Rapid  Input/Manipulation  Methods 

Tool  available=100 

Repetitive  Action  Capabilities  Method  available  but  takes  2  hands=75 

Y=100 

Align  Tool  N=0 

Y=100 

Reverse  Links  Tool  N=0 

Very  Consistent=100  (3  Ns  or  fewer  in  categories 
on  master  data  collection  sheet) 

Somewhat  Consistent=75  (5  Ns  in  categories  on 
master  uata  collection  sheet) 

Seldom  Consistent=25 
Not  Consistent=0 

Windows  Common  Look/Feel 
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Efficiency  Factors 

Comprehensive  Node  Representation 
Model  Description 


Rank  Definition 

Very  Consistent=100  (3  Ns  or  fewer  in  categories 
on  master  data  collection  sheet) 

Somewhat  Consistent=75  (5  Ns  in  categories  on 
master  data  collection  sheet) 

Seldom  Consistent=25 
Not  Consistent=0 
Y=100 
N=0 


Efficiency.  Factor  Results 


Table  21.  Results  for  Efficiency  Factors  Criteria 


Efficiency  Factors 

Rapid  Input/Manipulation 
Methods 

Analytica 

Netica 

Hugin 

BayesiaLab 

Repetitive  Action  Capabilities 

100 

100 

75 

100 

Align  Tool 

100 

100 

100 

0 

Reverse  Links  Tool 

0 

100 

100 

0 

Windows  Common  Look/Feel 
Comprehensive  Node 

75 

75 

100 

100 

Representation 

25 

50 

100 

100 

Model  Description 

0 

100 

0 

0 

gjNode  Edition 


All  of  the  applications  had  repetitive  action  capabilities  and  all  but 
BayesiaLab  had  an  align  tool;  both  capabilities  speed  model  building — important 
when  building  very  large  networks.  Netica  and  Hugin  supported  reversing  links, 
another  useful  capability  to  speed  the  network-building  process.  A  familiar 
Windows-like  environment  is  also  a  factor  that  speeds  task  completion;  all  tools 
were  acceptable,  but  Analytica  used  data 
field  screens  with  cells  and  dropdown 
boxes  for  some  of  its  data  entry  and  these 
were  harder  to  interpret  and  required  extra 
cursor  movements.  The  data  cells  were 
particularly  confusing,  as  the  default  view 
showed  a  single  cell  and  the  user  had  to 
select  that  cell  and  hit  the  Enter  key  to  add 
cells.  This  was  explained  in  the  tutorial, 
but  was  not  supported  in  the  data  screen 
itself.  Figure  19  shows  BayesiaLab’ s 
composite  dialog  box  that  allows  the  user 
to  rapidly  enter  all  supporting  node  data. 
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Figure  19.  BayesiaLab  composite  dialog 
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Error  Management  Evaluation  Criteria 


Table  22.  Definitions  for  Error  Management  Criteria 

Error  Management  Description 

Conditional  probability  errors  (failure  to  sum  to  1 )  and  cyclic 
Validation  Errors  errors  (node  connections  form  cycle). 

Program  warns  user  an  error  has  occurred  via  visible  or 
Warnings  audible  signals. 

Warning  includes  an  explanation  of  the  location  and  type  of 
Explanations  error. 

Program  can  be  set  to  repair  probability  errors  automatically 
Automatic  Repair  Function  (automatically  normalizes  data). 

Input  Errors  Errors  in  types  of  information  input  into  data  fields. 

Program  warns  user  an  error  has  occurred  via  visible  or 
Warnings  audible  signals. 

Warning  includes  an  explanation  of  the  location  and  type  of 
Explanations  error. 

System  &  Fatal  Errors  Warns  of  system  &  insufficient  memory  errors. 


Table  23.  Rank  Structure  for  Error  Management  Criteria 


Error  Management 
Validation  Errors 

Warnings 

Explanations 

Automatic  Repair  Function 

Input  Errors 

Warnings 

Explanations 

System  &  Fatal  Errors 


Rank  Definition 

Message  BoxdOO 

Symbol  w/popup=50 

Explains  probability  &  cycles=100 

Explains  cycles  only=50 

Must  set  to  normalized  00 

Warns  will  automatically  normalize=50 

Automatically  normalizes=0 

Boxed  text  warningdOO 
Symbol=75 

Message  pops  updOO 

Takes  2  hands  to  display  message=50 

Yd  00 

N=0 


Error  Management  Results 


Table  24.  Results  for  Error  Management  Criteria 


Error  Management 

Analytics 

Netica 

Hugin 

BayesiaLab 

Validation  Errors 

Warnings 

100 

100 

100 

100 

Explanations 

100 

100 

50 

50 

Automatic  Repair  Function 

50 

100 

0 

0 

Input  Frrnrs 

Warnings 

100 

100 

100 

75 

Explanations 

100 

100 

50 

50 

System  &  Fatal  Errors 

100 

0 

0 

0 
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This  set  of  criteria,  while  very  important  to  the  user,  was  not  thoroughly 
investigated.  Factors  involved  in  this  decision  included  the  variation  in  functions 
supported  by  the  applications  (e.g.,  Analytics  supported  dynamic  but  not  static 
cyclic  dependencies,  whereas  Netica,  Hugin  and  BayesiaLab  flag  and  prohibit 
directed  cyclic  dependencies)  and  the  lack  of  a  representative,  rigorous  test 
model.  The  standard  example,  “Asia”,  included  with  each  program,  didn’t  fully 
exercise  the  error  management  capabilities  of  the  systems. 

All  applications  prohibited  cyclic  errors  with  an  accompanying  explanation 
(, Analytical  warning  offered  an  alternative  means  of  defining  cycle  nodes  to 
make  the  cycle  dynamic).  Probability  table  error  management  was  handled  in 
three  different  ways:  Analytics  warned  that  it  would  normalize  the  illegal  entry  if 
directed  to  proceed  with  compilation,  Netica  simply  flagged  probability  table 
errors,  whereas  Hugin  and  BayesiaLab  automatically  normalized  values. 
Automatic  normalization  was  judged  a  fault  in  this  evaluation  because  it  denied 
user  control  and  hid  the  potential  introduction  of  erroneous  values  from  the  user. 

Input  errors  were  the  most  difficult  to  evaluate  because  the  evaluator,  with 

only  limited  knowledge  of  Bayesian  net 
building  and  a  limited  test  model,  did  not 
test  all  possible  error  conditions. 
Evaluation  in  this  section  was  based  on 
whether  the  system  provided  input  error 
notification,  how  readily  visible  the 
warning  was,  the  clarity  of  explanation 
and  provision  of  instructions  to  repair  the 
error.  All  systems  provided  some  form  of 
error  message  ( BayesiaLab  used  a 
warning  symbol  that  required  effort  to 
open);  none  provided  specific 
instructions  along  with  their  explanations. 
This  was  considered  a  serious  lack, 
given  the  complexity  of  the  network¬ 
building  task.  See  Figure  20  for  sample 
messages. 


Hugin  Error  Message  notes  an  evidence  input  error  for  a 
continuous  chance  node 


Analytica  Error  Message  notes  a  probability  input  error, 
offering  an  automated  fix  or  a  chance  to  edit  the  input. 


Figure  20.  Sample  error  messages 
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Visualizations  Evaluation  Criteria 


Table  25.  Definitions  for  Visualizations  Criteria 


Visualizations 
On  Screen 

Tables  &  Networks 

Graphs 

Text 

Error  Logs 

Printing 

Page-break  Preview 
Scale-to-Fit  Capability 
Completeness 


Description 

Displays  node  network  and  associated  conditional  probability  tables 
simultaneously. 

Displays  node  information  in  different  graph  formats  (bars,  line  graphs, 
etc. 

Displays  text  descriptions  of  node  network. 

Displays  user’s  session  error  log  in  text  box. 

Permits  the  modeler  to  see  where  the  page  breaks  will  occur  in  the 
network  or  table  and  adjust  page  setup  accordingly. 

Permits  the  modeler  to  scale  the  network  to  fit  the  page. 

Prints  nets,  tables,  graphs,  text  reports  and  error  logs  (or  some  subset). 


Table  26.  Rank  Structure  for  Visualizations  Criteria 


Visualizations 
On  Screen 
Tables  &  Networks 
Graphs 
Text 

Error  Logs 

Printing 

Page-break  Preview 
Scale-to-Fit  Capability 


Completeness 


Rank  Definition 


Each  of  the  On  Screen  attributes  rates  25  if  Y  and  0  if  N.  All 
four  attributes  sum  to  an  On  Screen  score  of  100. 


Y=100 

N=0 

Y=100 

N=0 

Prints  everything  it  shows=100  (includes  reports) 
Prints  tables,  network  &  text=75 
Prints  tables  &  network=50 


Visualizations  Results 


Table  27.  Results  for  Visualizations  Criteria 


Visualizations 

Analytica 

Netica 

Hugin 

BayesiaLab 

On  Screen 

Tables  &  Networks 

25 

25 

25 

25 

Graphs 

25 

25 

25 

25 

Text 

0 

25 

0 

25 

Error  Logs 

0 

25 

0 

0 

Printing 

Page-break  Preview 

100 

100 

0 

0 

GoqIo  to  Pit  Capability 

100 

0 

100 

mo 

Completeness 

100 

100 

50 

75 

50 


Figure  21.  Summary  visualizations  from  each  application 


Visualizations  were  divided  into  visual  representations  on  screen  and  print 
capabilities.  The  applications  varied  widely  in  the  representations  offered  and 
the  forms  taken.  In  Figure  21  above,  Analytics’ s  summary  visualization  was 
programmed,  while  the  others  were  available  as  menu  options.  The  quality  of 
the  visualization  varied  considerably.  Analytics  did  not  support  simultaneous 
representations  of  their  network  node  maps,  attribute  charts,  and  conditional 
probability  data.  In  contrast,  BayesiaLab  used  an  easily  read  and  understood 
composite  attribute/conditional  probability  table,  visible  with  the  node  map,  and 
Netica  even  displayed  the  network  description  and  error  log  in  frames.  The 
ability  to  add  a  network  description  to  aid  other  users  was  seen  only  in  Netica 
and  BayesiaLab ;  error  logs  (important  to  users  learning  the  system)  were 
available  only  in  Netica.  Print  capabilities  ranged  from  Hugin’s  restriction  to 
networks,  node  attribute,  and  probability  tables  only  to  Analytica  and 
BayesiaLab’ s  ability  to  print  all  visualizations  and  report  results.  Of  the  four 
applications,  only  Hugin  had  no  sensitivity  analysis  capability. 

4. 5. 9.4  AHP  Results 

As  noted  previously,  the  applications  varied  greatly  in  their  intended  use  as 
well  as  their  capabilities.  The  results  of  the  AHP  analysis  (Figures  22  and  23) 
showed  Netica  the  most  user  friendly  in  terms  of  user  support,  efficiency  of  entry, 
error  management,  and  supported  visualizations;  Analytica  was  a  close  second 
with  the  other  two  applications  ranking  significantly  lower  in  ease-of-use. 
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Figure  22.  Usability  Factors  Decision  Score 
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Figure  23.  Usability  Factors  by  Criteria 
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5  RECOMMENDATIONS 

5.1  Text  Mining  Tools 

As  mentioned  previously,  we  are  considering  three  categories  of  tools  -  Text 
Mining,  Investigative  Analysis,  and  Bayesian  tools  that  would  help  an  analyst  with 
his/her  task.  However,  as  the  emphasis  of  this  study  was  on  Bayesian  modeling 
tools,  we  have  not  had  a  chance  to  do  an  in-depth  evaluation  of  text  mining  and 
investigative  analysis  tools.  SRA,  a  regular  user  of  vendor  products,  has 
gathered  information  from  sources  that  do  in-depth  study  of  these  tools. 

For  text  mining  tools,  we  relied  on  market  research  done  by  SRA.  Table  28 
lists  well  known  tools  and  vendors  in  different  categories  of  text  mining. 


Table  28.  Text  Mining  Tools 


Technique 

Tool  Name 

Website 

Categorization  of 
documents 

Autonomy 

http://www.autonomy.com/Content/Technology/ 

Convera 

http://www.convera.com/Products/index.asp 

Entrieva 

http://www.entrieva.com/entrieva/index.htm 

InXight 

http://www.inxight.com/ 

Mohomine 

http://www.mohomine.com/ 

Stratify 

http://www.stratify.com/ 

Verity 

http://www.verity.com/products/index.html 

Summarization 

Copern  ic 

http://www.copernic.com/en/products/summarizer/index.html 

InXight 

http://www.inxight.com/ 

5.2  Social  Network  Analysis  Tools 

For  social  network  analysis  tools,  the  recommendation  is  based  on  the 
results  from  a  detailed  survey  done  for  MITRE  Corporation.  In  this  report,  i2’s 
Analyst  Notebook  Version  6  was  the  first  recommendation  and  the  next 
recommendation  was  Visual  Analytics  VisuaUnks.  For  further  details  regarding 
the  criteria  used,  other  tools  evaluated  and  generation  of  results,  refer  to  [3]  (end 
of  Section  4),  included  as  Attachment  8  to  this  report. 
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5.3  Modeling  Tools 
5.3.1  Performance 

From  Section  4.5.6,  it  can  be  seen  that  Hugin  ranks  as  the  optimal  Bayesian 
tool  among  all  the  ranked  alternatives.  The  next  best  alternatives  are  BayesiaLab 
and  Netica. 

Most  of  Hugin' s  high  decision  score  can  be  attributed  to  the  number  of 
features  it  supported  in  the  category  of  Input  Manipulation  and  the  fact  that 
criteria  Input  Manipulation  was  given  the  maximum  weight  towards  the  final  goal 
of  choosing  an  optimal  tool. 

BayesiaLab  has  excellent  design  sense  from  a  usability  standpoint.  Though 
there  are  some  “ease  of  use”  problems,  they  should  be  relatively  easy  to  fix. 

Both  Hugin  and  BayesiaLab  personnel  were  very  helpful  and  receptive  to  making 
changes  and  adding  features  in  response  to  specific  queries  and  customer 
requests  (in  fact  the  BayesiaLab  folks  have  already  made  some 
changes/updates  in  response  to  queries  from  this  study). 

Netica  and  BayesiaLab  representatives  are  already  addressing  some  of  the 
performance  issues  that  led  to  their  ranking  below  Hugin  (see  Table  12).  With 
the  addition  of  these  planned  extensions,  it  is  likely  that  the  performance 
rankings  for  Netica  and  BayesiaLab  will  improve. 

SIAM  is  the  only  GOTS  tool  that  we  analyzed  in  detail,  for  reasons  described 
in  Section  4.  One  of  the  major  drawbacks  of  SIAM  is  that  it  does  not  support 
Bayesian  Inferencing.  Other  features  that  SIAM  does  not  address:  the  learning 
of  probabilities  from  a  sample  set  and  learning  of  the  structure  of  the  net  from  a 
database.  These  features  are  useful  if  the  end  user  wants  the  tool  to  work  with 
existing  data.  In  Performance  &  Extensibility,  SIAM  ranked  low  because  of  the 
emphasis  placed  on  the  language  used  to  develop  the  computational  engine  and 
the  Graphical  User  Interface  (GUI).  The  preferred  language  was  C/C++  and 
SIAM  was  developed  in  Java.  The  lack  of  an  Application  Program  Interface  (API) 
for  SIAM  also  contributed  to  its  low  score.  All  that  said,  SIAM  does  have 
performance  features  that  will  be  useful  for  NASIC  in  the  near  term. 

Based  on  the  information  gathered  for  OCCAM,  it  can  be  said  that  it  seems 
to  be  addressing  the  domain  specific  needs  of  the  customer.  It  also  employs  all 
the  techniques  that  the  customer  is  planning  to  apply  to  build  a  decision  support 
system  that  helps  model  the  influences  impacting  human  behavior. 

Sample  might  be  useful  to  NASIC/BPB  if  they  plan  to  employ  multiple 
artificial  intelligence  techniques  (in  addition  to  Bayesian  Nets). 

Although  Metrica’s  PSYOP  DSS  tool  is  not  applicable  as  one  of  the  ranked 
alternatives  (does  not  use  Bayesian  methods),  the  feature  of  building  a  database 
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that  contains  data  supporting  PSYOP  objectives  (and  helping  the  user  based  on 
this  information)  is  a  very  useful  technique.  This  technique  can  be  applied  to 
Bayesian  tools  to  help  the  user  determine  the  a-priori  probabilities. 

5.3.2  Ease  of  Use 

Where  Hugin  and  BayesiaLab  were  the  top  modeling  tools  from  a 
performance  perspective,  Netica  and  Analytica  stood  out  from  those  two  in 
usability.  Netica  rated  most  “user  friendly”  in  terms  of  user  support,  efficiency  of 
entry,  error  management,  and  supported  visualizations.  Analytica  was  a  close 
second  in  all  these  categories,  with  the  Hugin  and  BayesiaLab  ranking 
significantly  lower  in  ease  of  use. 

SRA  continues  to  receive  information  from  all  these  companies.  As  we  get 
more  information,  we  will  update  our  AHP  analysis  to  ensure  we  have  accurately 
modeled  them  and  get  the  correct  relative  ranking.  As  we  get  results,  we  will 
provide  them  to  the  customer.  However,  from  both  a  performance  and  a  usability 
perspective,  it  would  be  far  preferable  to  validate  results  from  this  study  against  a 
representative  sample  problem  or  scenario. 

5.3.3  Cost 

Table  29  summarizes  the  cost  per  single  license,  quantity  licenses,  and  site 
license  of  the  top  five  modeling  tools.  As  some  of  the  tools  prices  are  listed  in 
Euros,  refer  to  http://www.x-rates.com/calculator.html  for  an  equivalent  price  in 
Dollars. 


Table  29.  Cost  Summary. 


Tool 

Pricing  Structure 

(GUI  and  API  Priced) 

Hugin 

1  License  -  6300  Euros 

2  Licenses  -  8390  Euros 

5  Licenses  -  10,465  Euros 

10  Licenses  -  12,550  Euros 

Site  License  -  16,785  Euros 

(GUI  and  API  Priced) 

BayesiaLab 

1  License  -  3450  Euros 

5  Licenses  -  1 0,350  Euros 

10  Licenses  -  17,250  Euros 

50  Licenses  -  43,125  Euros 

Netica 

GUI  -  $585  per  License,  API  -  $685 

Note  1 :  Site  licenses  available  for  5  times  as 
much. 

Note  2:  API  embedded  is  $20  to  $175 
depending  on  the  volume. 
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Tool 

Pricing  Structure 

Analytica 

Professional  -  $1295 

Enterprise  -  $2495 

Analytical  Decision  Engine  -  $6000 

Note  1 :  Website  (Attachment  3)  gives 
comparison  of  features  of  these  different 
versions) 

Note  2: 15%  discount  for  10+  Licenses 

Note  3:  25%  discount  for  50+  Licenses 

SIAM 

All  US  gov't  agencies  hold  a  license  to  use 

SIAM  for  gov't  purposes  (just  need  federal  ID) 

Complete  cost  information  for  the  rest  of  the  evaluated  tools  can  be  found  in 
the  “General  Features”  worksheet  in  Attachment  3. 

/  6  CONCLUSION 

6.1  NASIC’s  Investment  in  Analytical  Capabilities 

Many  viable  text/data  mining  options  are  available  to  NASIC/BPB.  SRA  is 
very  familiar  with  NetOwl,  and  has  used  it  for  a  variety  of  applications.  However, 
other  options  were  presented  in  Section  5. 

From  discussions  with  experts  in  the  field  of  Social  Network  Analysis,  we 
were  referred  to  a  Mitre  Corporation  report  (Attachment  9)  that  ranked  i2’s 
Analyst  Notebook  Version  6  at  the  top  of  the  list  of  these  capabilities.  The  next 
best  capability  recommendation  was  Visual  Analytics  VisuaLinks. 

Based  on  the  results  of  this  study,  NASIC/BPB  has  at  least  five  viable 
options  to  meet  mission  modeling  requirements:  Hugin,  BayesiaLab,  Netica, 
Analytica,  and  SIAM.  Unfortunately,  the  tool  that  ranked  best  in  terms  of 
performance  (Hugin)  is  also  rated,  relative  to  the  other  tools,  more  difficult  to  use. 
BayesiaLab,  another  highly  ranked  tool  in  terms  of  performance,  is  also  ranked 
lower  in  “ease  of  use.”  Netica  and  Analytica,  while  easier  to  use,  did  rank  lower  in 
performance.  The  lack  of  a  clear  “winner”  is  further  complicated  by  the  fact  that 
some  extensions  to  the  performance  capabilities  of  both  Netica  and  BayesiaLab 
are  planned  in  the  next  release  of  these  tools.  None  of  these  options  is  out  of  the 
running  in  terms  of  cost/price,  and  (of  the  five  top-rated  performance  options) 
SIAM  has  the  advantage  of  being  available  for  licensing  free  of  charge  to 
government  customers. 

On  balance,  especially  if  ease  of  use  is  a  critical  factor  for  NASIC  and  the 
customer  is  willing  to  wait  for  the  performance  improvements  (Table  12)  planned 
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in  the  next  release,  Netica  becomes  a  very  attractive  option.  See  Table  30  below 
for  a  summary  of  performance,  usability  and  cost  results. 


Table  30.  Project  Results  Summary 


Performance 

Usability 

Cost 

GF 

IM 

PE 

10 

Overall 

Performance 

US 

EF 

EM 

VR 

Overall 

Usability 

Per  10 
Licenses 

Analytica 

G 

Y 

B 

B 

G 

G 

G 

G 

G 

G 

551,000* 

BayesBuilder 

G 

R 

G 

R 

Y 

NR 

NR 

NR 

NR 

NR 

4000  Eu 

($4,800)+ 
per  seat 

BayesiaLab 

/ 

B 

G 

G 

B 

Y 

f 

Y 

< 

i 

17,250  Eu 

($20,700)+ 

Dxpress 

G 

/ 

B 

R 

Y 

NR 

NR 

NR 

NR 

NR 

Provided  at 
negotiation 

Ergo 

/ 

R 

B 

t 

t 

NR 

NR 

NR 

NR 

NR 

? 

Hugin 

G 

B 

G 

G 

B 

f 

G 

Y 

( 

f 

12.550  Eu 

($15,060)+ 

Netica 

G 

f 

B 

B 

G 

G 

B 

G 

G 

B 

$6,350** 

SIAM 

B 

f 

1 

G 

G 

NR 

NR 

NR 

NR 

NR 

Free 

Bnet2000 

R 

Y 

G 

G 

( 

NR 

NR 

NR 

NR 

NR 

Not 

provided 

Notes: 

B  =  Best  relative  to  others  G  =  Acceptable  f  =  Marginal  R  =  Worst  relative  to  others 
*  ADE  version,  assumes  (single  seat  price  XI 0)  minus  15%  discount  for  10+  licenses 
**  Price  for  site  license  (no  price  given  for  10  seats) 

+  Conversion  to  dollars  assumes  1 .2  dollars  per  Euro  (approximate  current  exchange  rate) 

If  NASIC/BPB  wants  further  verification/validation  of  the  results  of  this  study 
before  making  an  investment  in  a  particular  modeling  capability,  SRA 
recommends  a  more  comprehensive  evaluation  using  a  sample  or  “benchmark” 
problem. 

6.2  Study  Limitations 

There  are  some  limitations  on  how  these  results  should  be  used.  An 
important  factor  when  interpreting  the  results  is  to  realize  that  the  rankings  for  the 
tools  were  not  generated  based  on  building  a  Bayesian  net  using  all  the  tools  and 
then  comparing  the  features.  The  ranks  were  primarily  based  on  the  responses 
from  the  tool  vendors,  information  from  the  trial  versions  of  the  tools,  and 
information  on  the  web.  In  this  process  we  might  have  overlooked  some  criteria 
that  might  prove  useful  when  building  a  Bayesian  model.  Also,  the  final  decision 
scores  of  the  tools  are  a  result  of  the  criteria  chosen  and  the  weights  assigned  to 
them.  The  decision  scores  can  change  if  a  user  evaluates  the  tools  with  a 
different  set  of  criteria  and  different  weights. 
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Another  point  to  consider  is  that  the  SRA  team  worked  with  demonstration 
versions  of  these  tools  as  they  exist  today.  As  planned  extensions  are  added, 
and  as  more  information  is  made  available,  these  updates  should  be  captured 
and  considered  in  the  analysis  before  finalizing  an  investment  decision. 

6.3  Recommended  Follow-On  Activity 

SRA  team  members  recommend  further  work  to  develop  and  implement  a 
“benchmark”  evaluation  problem  or  scenario  to  verify  the  results  of  this  study.  As 
discussed  above,  tool  rankings  can  be  sensitive  to  both  criteria  and  weighting.  A 
sample  problem  or  scenario  that  is  representative  of  how  the  tools  will  actually  be 
used  by  NASIC  analysts  can  provide  an  additional  level  of  insight  and  confidence 
to  validate  (or  revise)  these  results. 

6.4  Interest  from  Industry  and  Government 

Many  contacts  (see  Attachment  10)  are  interested  in  sharing  the  results  of 
this  effort. 
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7  Attachments 


1.  Requirements  Trace,  Influence  Operations  and  PEBO 
Architectures 

2.  Analyst  Questions 

3.  Project  Criteria/Data  Sheets 

4.  Definition  of  Terms  and  Tutorial  Briefing 

5.  Prioritization  Briefing 

6.  Market  Survey  Questionnaire 

7.  Questionnaire  Response  Raw  Data 

8.  The  Mitre  Corporation  Report 

9.  Significant  Evaluation  Issues  and  Ease  of  Use  Analysis 

10.  POCList 

11.  Project  Glossary 
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Attachment  1 


Requirements  Trace,  Influence  Operations 
and  PEBO  Architecture 
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[REQUIREMENTS  TRACE  FROM  CAPABILITIES  TO  CRITERIA- 
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Attachment  2 


Analyst  Questions 
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MEMORANDUM  FOR  RECORD 


SUBJECT:  12  Sep  03  Analytical  Capability  for  Behavioral  Influence  Operations  Team 
Questions  for  Customer  (Influence  Operations  Planners,  Analysts) 

TO:  NAIC/BPB  (Eric  Braeden) 

1 .  Eric,  here  is  our  first  crack  at  a  list  of  questions.  Hopefully,  this  will  continue  our 
Task  1  progress  toward  a  full  understanding  of  how  planners  and  analysts  will  use  the 
tools.  This  will  ensure  we  have  the  right  criteria  when  we  start  evaluating  candidates 
and  alternatives  in  Tasks  2,  3,  and  4. 

2.  Questions  focused  on  the  initial  planning  phase: 

2.1.  Describe  your  planning  cycle  for  a  representative/relevant  RFI  or  problem. 

2.1.1.  Who  is  basic  Customer  set  and  what  do  they  typically  ask  you  to  provide? 

2. 1 .2.  What  Official  Documents,  TTP,  Regs  and  checklists  do  you  refer  to  for 
guidance? 

2.1.3.  Do  you  use  any  worksheets  for  PSYOP  planning  and  research? 

2.1.4.  What  Information  Sources  do  you  use?  Do  you  use  and  have  access  to  the 
following  Databases  and  programs? 

2. 1.4.1. Special  Operations  Command,  Research,  Analysis  and  Threat 
Evaluation  System  (SOCRATES) 

2. 1.4.2.  Psychological  Operations  Automated  Data  System  (POADS) 

2.1.4.3. Foreign  Publication  Procurement  Program  (FPPP) 

3.  Questions  focused  on  the  analysis  process: 

3.1.  Describe  your  PSYOP  Target  Audience  analysis  processes 

3.1.1.  Type  and  Process 

3. 1.1.1.  Value  analysis:  Quantitative  and  Qualitative 

3.1.1.2. Nodal  analysis 

3.1.2.  What  factors  are  taken  into  consideration? 

3.1.3.  What  are  your  target  selection  criteria? 

3.1.4.  How  do  your  targets  get  nominated  and  approved? 

3.1.5.  How  do  you  analyze  secondary  audiences  for  effects? 

3.1.6.  What  are  a  typical  Timeframes  for  RFI’s,  process  and  production? 

3.1.7.  Do  you  track/monitor  PSYOP  operations?  How?  (i.e.,  PSYOP  Campaign 
Control  schedules  with  PSYOP  intensity,  objectives,  temporal  schedules) 

3.1.8.  Do  you  currently  conduct  a  form  of  “PSYOP  weaponeering  for  “targets”? 

3.1.9.  Is  environmental  analysis  taken  into  consideration  and  used?  (Climate, 
Weather,  and  Geographic) 

3.1.10.  How  are  “targets”  tracked  in  databases?  (~  BE  numbers) 

3.1.11.  What  type  of  role  do  you  play  in  execution  planning  if  any  at  all? 

3.1.12.  Do  you  conduct  PSYOP  Effectiveness  Assessment? 

3. 1 . 12. 1 .  Are  MOEs  determined  mid  iiicasuicd?  Hu w? 

3.1.1 2.2.  Are  Objectives  not  met  by  MOEs  incorporated  into  future 
planning,  and  PSYOP  assessment? 
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4.  Questions  that  dig  into  what  cognitive,  cultural,  organizational  characteristics  are 

valuable  to  the  analyst: 

4. 1 .  Points  on  decision  making  style,  group  goals  and  organizational  behavior  were 
noted;  however,  is  it  useful  to  include  other  culturally-based  influences  in 
cultural  models  used  to  test  software  capabilities?  Should  primary  focus 
emphasize  any  one  of  the  following? 

4.1.1.  Cognitive  characteristics?  (e.g.,  idiocentric  vs.  allocentric  orientation, 
mastery  vs.  fatalistic  orientation,  past  vs.  present  vs.  future  time  orientation, 
etc.) 

4. 1 .2.  Institutional  characteristics?  (e.g.,  Muslim,  largely  agricultural/pastoral, 
tribal  affiliations,  oil-based  international  economy  with  wealth  clustered  in 
ruling  family,  etc.) 

4.1.3.  Sensitivities?  (e.g.,  gender  roles,  caste/class  systems,  social  position  of 
children/elderly,  social  taboos,  etc.) 

4.1.4.  Other? 

4.2.  How  do  you  incorporate  cultural  knowledge  into  your  planning  now? 

4.2. 1 .  Is  culture  a  critical  thread  in  all  influence  planning,  a  means  of  fine-tuning 
non-cultural  influences,  the  focal  point  of  some  planning  efforts,  etc? 

4.2.2.  Are  cultural  issues  always  specifically  included  in  influence  operations? 

4.3.  How  do  you  foresee  incorporating  culture  in  the  future? 

4.3.1.  Do  you  expect  to  increase  employment  of  cultural  influences? 

4.3.2.  Do  you  need  cultural  matrices  that  map  associated  strains  of  cultural 
influences  and  relate  specific  cultural  influences  to  specific  groups? 

4.3.3.  Are  there  plans  to  collect  specific  data  on  effectiveness  of  cultural 
influence  employment? 

4.3.4.  Are  there  plans  to  try  to  separate/evaluate  the  relative  effects  of  specific 
circumstances  on  effectiveness? 

4.4.  Can  you  give  several  examples  in  which  culture  played  a  role  in  NAIC  product 
development? 

4.4.1.  What  are  some  representative  cases? 

4.4.2.  What  are  some  atypical  cases? 

5.  Questions  focused  on  current  tools  that  analysts  use  or  are  aware  of: 

5.1.  What  type  of  Products  do  you  currently  produce? 

5.1.1.  What  are  typical  NAIC  products  like  in  format?  (e.g.,  Army  PSYOP 
forms,  multi-page  text  reports?) 

5. 1 . 1 . 1  .How  might  cultural  influences  be  incorporated  in  that  format?  (e.g., 
specifically  identified,  fully  justified,  or  transparent  to  the  user) 

5.1.2.  What  software  do  you  use  currently  developing  your  products? 

5 . 1 .2. 1  .What  works  for  you? 

5. 1.2.2. What  doesn’t  work  for  you? 

5.1.3.  What  non-software  supported  techniques  do  you  use  currently  to  develop 
products? 

3. 1 .3. 1  .What  capabilities  wuuld  have  helped  )uu  piuduee  tliuse  pic/duets? 

5.2.  What  kinds  of  data  base  management  systems  are  used  now? 
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5.3.  What  is  the  nature  of  data  that  is  useful  to  you  now?  (e.g.,  text,  images,  video, 
audio,  etc.)  Any  expectations  that  this  will  change  in  the  future? 

5.4.  Are  data  visualization  tools  used? 

5.5.  What  types  of  decision  modeling  tools  are  used? 

5.6.  What  type  of  tools  would  help  you  better  perform  PSYOP  analysis? 

6.  We  will  be  pursuing  the  answers  to  these  questions  through  interviews  with  NAIC 
and  other  Influence  Operations  analysts  over  the  next  two  weeks.  Any  assistance  you  can 
provide  will  be  greatly  appreciated. 


Michael  L.  Zywien,  Principal 
Project  Technical  Lead 


cc:  Greg  Jannarone 

Capt  Tim  Gameros 
Larry  Daniel 
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Attachment  3 


Project  Criteria/Data  Sheets 
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Attachment  4 


Definition  of  Terms  and  Tutorial  Briefing 


74 


Terms  that  keep  coming 
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Terms  in  Detail 


o 

-  o) 


0)  ^ 

c  - 
o  tj 
U  c 

I 


1 

•  MM 

_Q 

a 

o 

ft) 

>^-Q 
£-  O 

i— 

O 

ft) 

Q_ 

> 

i- 

O 

•  MM 

• 

5 

+- 

ft) 

•  MM 

•  MM 

-Q 

V) 

o 

o 

a 

n 

V) 

JD 

o 

-o 

•  MM 

o 

ft) 

Q_ 

a 

d 

o 

c 

o 

a 

N 


X 

a 


•+— 

^  > 

V)  — 

a  a 

~o  E  c 
S  ft)  ° 

.O  “a  o 
a)  a  ^ 


o  .b  .t 


V) 


o 

V) 

4-  fc) 

^  >  fO 

'  +Z  N 

o  o  •— 

r-  ?  £ 


“a 

i 


x 

a 

e 


77 


Bayesian  Network 


V) 

fc) 

o  £ 
V  5) 


-Q 

a 

JD 

o 


fc) 

"U  ^ 
fc) 


<u  a 
v)*r 

?>  a 


o  co 

T5 
fc) 

V)  -t- 

8-0 

-h-  V) 
V) 


fc) 

o 


"D 


fc) 

"D 

V) 

•  MB 

V) 

X) 

o 


fc) 

fc) 


e 

o 


<^) 

fc) 

£ 


O  fc) 
VJ  OQ_Q 


fc)  — 
■°"D 

C  t3 

<tf  fc) 


(0 

<\) 


78 


Every  node  is  associated  with 
Conditional  Probability  Table  (CPT) 

•  CPT  represents  information  about  the  effect  of 
parent  nodes  on  the  child  node. 


Populating  CPTs 
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Influence  Diagrams 
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Example  of  Influence 
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Decision  Trees 
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Criteria  model 
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Criteria  Model  Example 
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Evaluating  Networks 
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-  Conditioning 

-  Elimination 

-  Stochastic  Sampling 
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Summary 
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Summary  Contd 
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\\Dayton2\Projects\NAIC  Infulence 


Greg's  Macro  Criteria 
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Attachment  5 


Prioritization  Briefing 
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Overview 
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Discussion  -  Feedback 
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Task  5  (Demonstrations)  eliminated 

Task  6  (Final  report)  -  Draft  outline  available 
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Analyze  data 

Report  results  to  customer 


Macro  Criteria 
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Elements  of  Decision  Analysis 
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•Variable  Elimination  -Value  Trees 


Questionnaire  mapped  to  Excel  Data  Capture  Sheet 
—  Questionnaire  mapped  to  spreadsheet 
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Market  Survey 
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38  relevant  research  efforts 
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—  Hugin  Expert 

Many  contacts  are  interested  in  sharing  results  of  this  effort 


108 


Attachment  6 


Market  Survey  Questionnaire 


Tool  Name: 

Website: 

Company  Name: 

General  Features 

1.  Ease  of  use:  (B) 

■  Is  there  feedback  for  processing  delays? 

■  Do  menu  items  and  text  descriptions  use  language  that  is  commonly  understood? 

■  Are  menu  items  in  standard  places?  Are  layouts,  terms,  icon  screen  positions,  windows,  dialog  boxes  consistent? 

■  How  many  steps  does  it  take  to  perform  a  frequently  used  action  (desire  not  buried  in  menus;  not  multi-step)? 

■  Do  you  show  user  that  he/she  has  made  a  mistake?  How  is  user  aided  in  correcting  mistakes? 

■  Do  you  visually  display  system  objects,  actions,  and  options  to  users  or  are  these  available  textual ly?  How  do  you 
aid  users'  memory  for  these  choices?  Are  instructions  easily  available  if  the  user  forgets  or  loses  his/her  way? 

■  Does  your  system  cater  to  novice  and  expert  users?  Allow  users'  to  create  shortcuts  for  frequently  used  actions? 

■  How  is  help  and/or  documentation  provided  to  the  user? 

■  How  does  your  software  handle  making  best  use  of  limited  screen  “real  estate”? 

■  Top  5  “trouble”  reports? 

2.  Cost:  Seat  license;  upgrades;  support;  maintenance;  (for  government  customer)(C) 

3.  Maturity  of  the  tool:  Number  of  years  in  the  market,  #  versions/upgrades.  (D) 

4.  Targeted  Industries:  Business,  medical,  govt.,  etc  (E) 

5.  Major  clients:  (E) 

6.  Type  of  Operating  system  supported:  List  all  applicable  OS  (F) 

7.  Recommended  Hardware:  (list  both  minimal  and  optimal)  (G) 

a.  Processing  power: 

b.  RAM: 

c.  Other: 

8.  Architecture  support:  (H) 

a.  Client  -  Server  Model:  Yes  I  I  No  I  I 

b.  Desktop:  Yes  I  I  No  I  I 


no 


Input  Manipulation 


1 .  T ype  of  Network  support: 

a.  Bayesian  Networks:  Yes  O  No  Q  (B) 

b.  Decision  Networks/Influence  Diagrams:  Yes  Q  No  HD  (C) 

c.  Both  Bayesian  Nets  and  Influence  Diagrams:  Yes  O  No  O  (D) 

d.  Decision  Trees:  Yes  Q  No  O  (C) 

2.  Model  Building  capability: 

a.  Using  GUI:  Yes  □  No  □  (E) 

b.  Other:  Using  a  custom  language,  batch  mode  etc  (F) 

c.  From  databases  (i.e.  Structure  learning):  Yes  Q  No  Q  (G) 

3.  Type  of  graph  support: 

a.  Only  Undirected  graph:  Yes  Q  No  O  (H) 

b.  Only  Directed  graph:  Yes  [~~l  No  I  I  (D 

c.  Chain  graphs  (i.e.  mixed  undirected  and  directed):  Yes  H3  No  Q  (J) 

4.  Conditional  Probability  Table: 

a.  Specify  Tables:  Yes  O  No  CH  (K) 

b.  Specify  Equations:  Yes  O  No  [H  (L) 

c.  From  databases  (learn  from  training  sets):  Yes  □  No  Q  (M) 

d.  Other:  Other  Input  techniques  used  (N) 

5.  Validity  of  the  model: 

a.  Consistent  Probability  check  of  variables  in  the  net:  Yes  O  No  O  (O) 

b.  Cyclic  Dependency  structure  check  among  variables:  Yes  Q  No  CD  (P) 

6.  Inference  Algorithm  used:  List  all  relevant  algorithms  (Q) 

7.  Analysis  techniques  provided/supported:  List  and  explain  (R) 


8.  For  tools  using  Bayesian  Networks  (Ref.  Col  B  data)  are  there  built  in  methods  to 
trim  the  network  or  otherwise  limit  computational  complexity  with  “relevance 
reasoning”  or  any  other  method:  Yes  Q  No  Q  (S) 

a.  If  yes,  explain:  (S) 

Performance  and  Extensibility 

1 .  Language  used  to  develop: 

a.  GUI:  (B) 

b.  Computational  Engine:  (C) 

2.  Is  source  code  generally  available  to  a  government  end  user:  Yes  Q  No  Q  (D) 

(If  no,  is  there  any  circumstance  where  the  source  code  could  be  made  available?) 

3.  Is  the  API  available  to  the  end  user:  Yes  D  No  O  (E) 

4.  Availability  of  the  tool’s  performance  measurement:  Yes  Q  No  Q  (F) 

a.  If  yes,  please  indicate  the  benchmark/procedure  used  to  measure  performance: 

5.  Maximum  Nodes  a  single  model  can  support:  Max  nodes  a  model  can  support  when 
running  on  the  recommended  system  (G) 

6.  Error  recovery  support  (if  something  crashes,  what  happens?):  (H) 
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Interoperability 


1.  Ability  to  interface  with  databases:  Yes  O  (B,C)  No  O 

a.  If  Yes, 

i.  Using  Open  Data  Base  Connectivity  (ODBC):  Yes  Q  No  O  (C) 

1 .  If  Yes,  list  the  data  bases  the  ODBC  driver  supports: 

ii.  Databases  it  can  interface  directly  with  (List  them):  (B) 

b.  Does  it  support  SQL  interface:  Yes  O  No  Q  (D) 

c.  Can  models  created  in  the  tool  be  saved  to  a  database:  Yes  Q  No  Q  (E) 

2.  Format  used  to  the  save  the  model:  (E) 

3.  Export  the  model  to  other  applications:  Yes  O  No  O  (F) 

a.  If  yes,  list  the  applications  the  model  can  be  exported  to:  (F) 

4.  Ability  to  export  results/graphs  generated  from  the  “what-if  ’  scenarios  or  the 
analysis:  Yes  Q  No  D  (G) 

a.  If  Yes,  list  applications  that  it  can  be  exported  to:  Eg:  PPT,  Word,  JPEG,  PDF 
format,  PS  format,  etc  (G) 

5.  Tools  uses  open  standards:  Yes  Q  No  Q  (H) 

a.  If  yes,  explain:  (H) 

6.  Provide  group  collaboration  on  model  building  and  analysis:  Yes  HH  No  Q  (I) 

(e.g.,  can  nodes  be  added  as  “sub-models”  created  by  another  user?) 
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Attachment  7 


Questionnaire  Response  Raw  Data 
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Tool  Name:  Analytica 
Website:  www.lumina.com 

Company’s  Name:  Lumina  Decision  Systems,  Inc 

General  Features 

1.  Cost:  Professional  $1295,  Enterprise  $2495,  ADE  $6000  (see  Web  site  for 
lower  prices  until  Nov  21  2003).  15%  discount  on  10+  licenses  25%  discount 
on  50+  licenses. 

2.  Maturity  of  the  tool:  Mature.  First  released  1996,  now  in  release  3.0 

3.  Targeted  Industries:  General:  including  technology,  aerospace,  consumer, 
defense,  healthcare,  energy  &  environment,  higher  education 

4.  Major  clients:  Thousands.  See  www.lumina.com/ana/customers.htm 

5.  Type  of  Operating  system  supported:  Windows  95, 98,  NT  4.0, 2000,  XP,  Mac 
OS  up  to  9.2 

6.  Recommended  Hardware: 

a.  Processing  power:  Pentium  or  later,  or  Power  PC 

b.  RAM:  64MB 

c.  Other:  Color  XGA  screen 

7.  Architecture  support: 

a.  Client  -  Server  Model:  Yes  [X]  No  I  I 

b.  Desktop:  Yes  f>^]  No  I  I 

Additional  Comments:  Client-server  model  supported  by  Analytica  Decision 
Engine.  User  must  write  end-user  code.  Analytica  Web  Publisher  with  automatically 
generated  Web-based  client  user-interface  expected  release  Q1  2004. 
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Input  Manipulation 


1 .  Type  of  Network  support: 

a.  Bayesian  Networks:  Yes  HU  No  1  I 

b.  Decision  Networks/Influence  Diagrams:  Yes  HU  No  HU 

c.  Both  Bayesian  Nets  and  Influence  Diagrams:  Yes  ^  No  HU 

2.  Model  Building  capability: 

a.  Using  GUI:  Yes  ^  No  □ 

b.  From  databases(i.e.  Structure  Learning):  Yes  I  I  No  HU 

c.  Other: 

3.  Type  of  graph  support: 

a.  Only  Undirected  graph:  Yes  HU  No  HU 

b.  Only  Directed  graph:  Yes  HU  No  I  1 

c.  Chain  graphs  (i.e.  mixed  undirected  and  directed):  Yes  I  1  No  I  I 

4.  Conditional  Probability  Table: 

a.  Specify  Tables:  Yes  HU  No  I  I 

b.  Specify  Equations:  Yes  HU  No  1  I 

c.  From  databases  (learn  from  training  sets):  Yes  I  I  No  HU 

d.  Other:  Can  provide  general  functional  form  of  relations,  including 
procedures  to  generate  probability  distributions 

5.  Validity  of  the  model: 

a.  Consistent  Probability  check  of  variables  in  the  net:  Yes  HU  No  HU 

b.  Cyclic  Dependency  structure  check  among  variables:  Yes  ^  No  HU 

6.  Inference  Algorithm  used:  Monte  Carlo  and  Latin  Hypercube 

7.  Analysis  techniques  provided:  Many,  including  parametric  sensitivity, 
correlation  importance  analysis,  optimization.  Note  that  Analytic  can  handle 
dynamic  models  (variables  changing  over  time)  without  having  to  replicate 
nodes  for  time  period.  This  makes  it  vastly  easier  to  build,  faster  to  evaluate, 
and  simpler  to  understand  dynamic  models  than  competing  methods. 
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8.  For  tools  that  support  Bayesian  Networks  (Ref.  Question  1)  are  there  built  in 
methods  to  trim  the  network  or  otherwise  limit  computational  complexity  with 
“relevance  reasoning”  or  any  other  method:  Yes  I  I  No  [Xl 
a.  If  yes,  explain: 

Additional  Comments:  Analytica  is  designed  for  and  extremely  versatile  and 
efficient  for  forward  or  causal  reasoning  with  influence  diagrams.  It  is  limited 
for  backward  or  diagnostic  reasoning. 

Performance  and  Extensibility 

1 .  Language  used  to  develop: 

a.  GUI:  C++ 

b.  Computational  Engine:  C++,  Analytica  scripting 

2.  Is  source  code  available  to  the  end  user:  Yes  I  I  No  [Xl 

3.  Is  the  API  available  to  the  end  user:  Yes  fx]  No  I  I 

4.  Availability  of  the  tool’s  performance  measurement:  Yes  O  No  ^ 

a.  If  yes,  please  indicate  the  procedure  in  measuring  the  performance: 

5.  Maximum  Nodes  a  single  model  can  support:  Tens  of  thousands,  each  with 
multidimensional  values,  comparable  to  millions  for  conventional 
representations 

6.  Error  recovery  support:  Yes  [Xl  No  I  I 

a.  If  yes,  explain:  Extensive  diagnostic  and  error  handling,  including  the 
ability  to  continue  execution  with  partially  undefined  or  infinite 
values.  Interpretive  mode  makes  it  easy  to  find  problems.  Execution 
profiling  (in  Enterprise  version)  supports  model  optimization. 
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Additional  Comments:  Analytica  provides  its  own  general  scripting  language, 
which  is  very  powerful  especially  for  multidimensional  modeling.  Some  users 
who  want  to  do  substantial  diagnostic  inference  use  Analytica  UI  to  create 
models  and  export  to  Winbugs  to  evaluate  them. 

Interoperability 

1 .  Ability  to  interface  with  databases:  Yes  ^  No  O 

a.  If  Yes, 

i.  Using  ODBC:  Yes  Kl  No  □ 

ii.  Databases  it  can  interface  directly  with  (list  them):  Any  that 
supports  ODBC 

b.  Does  it  support  SQL  interface:  Yes  [X]  No  1  I 

c.  Can  the  models  be  saved  to  a  database:  Yes  [X]  No  O 

2.  Format  used  to  the  save  the  model:  XML  file  format 

3.  Export  the  model  to  other  applications:  Yes  ^  No  □ 

a.  If  yes,  list  the  applications  the  model  can  be  exported  to:  Databases  and 
editors  that  support  XML.  Model  may  also  be  edited  in  any  word 
processor. 

4.  Ability  to  export  results/graphs  generated  from  the  “what-if  ’  scenarios  or  the 
analysis:  Yes  E><]  No  I  I 

a.  If  Yes,  list  applications  that  it  can  be  exported  to:  Can  copy  and  paste,  or 
OLE  hotlink  tables  and  graphs  to  Microsoft  Excel,  Powerpoint  and 
many  other  common  Windows  applications.  Can  use  Excel  seamlessly 
for  graphing  results.  Can  save  table  results  into  tab-delimited  files  for 
viewing  and  analysis  in  Excel  or  most  statistical  and  database 
packages. 

5.  Tools  uses  open  standards:  Yes  [>\1  No  I  I 
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6.  Provide  group  collaboration  on  model  building  and  analysis:  Yes  ^  No  I  I 

Additional  Comments:  Multiple  users  can  collaborate  in  building  models,  and  can 
distinguish  inputs  and  scenarios  by  multiple  users  or  experts  by  indexing. 

Additional  Notes:  Each  variable  or  node  in  Analytica  can  represent  a  single 
value  (number  or  text)  or  a  multidimensional  table.  This  facility,  along  with  its 
Intelligent  Array  capabilities  provide  great  convenience,  flexibility,  and  speed 
for  scaling  up  to  handle  large  "industrial-scale"  problems. 

See  www.lumina.com/casestudies  for  sample  applications 
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Tool  Name:  BayesBuilder 

Website:  http://www.mbfys.kun. nl/snn/nijmegen/index.php3?page=31 
Company’s  Name:  Smart  Research  BV 

General  Features 

1.  Cost:  2000  Euro  per  license  for  GUI  only.  4000  Euro  per  license  including 
API 

2.  Maturity  of  the  tool:  4  years 

3.  Targeted  Industries:  BayesBuilder  was  constructed  for  a  research  project  that 
aimed  to  build  a  large  scale  model  for  medical  diagnostics. 

4.  Major  clients:  Our  main  commercial  client  is  SKF.  They  used  BayesBuilder+ 
API  to  build  a  web-based  diagnostic  system  for  bearing  failure  analysis.  See 
http://evolution.skf.com/gb/article.asp?articleID=471 

5.  Type  of  Operating  system  supported:  Win32  Release  for  Windows  95,  Windows 
98  and  Windows  NT  (4.0)  -  or  higher  -  on  Intel  hardware. 

6.  Recommended  Hardware: 

a.  Processing  power:  Pentium  or  faster 

b.  RAM:32  megabytes 

c.  Other:  22  Mb  on  harddisk;  Minimum  desktop  areafor  GUI:  800x600 
pixels.  Color  palette:  256  colors,  VGA. 

7.  Architecture  support: 

a.  Client  -  Server  Model:  Yes  [X]  No  Q 

b.  Desktop:  Yes  ^  No  I  I 

Additional  Comments:  for  client  server  model,  one  needs  to  purchase  the  API  (  a 
library  of  JAVA  calls).  API  runs  independently  of  GUI. 
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Input  Manipulation 


1 .  T ype  of  N etwork  support : 

a.  Bayesian  Networks:  Yes  13  No  l~~l 

b.  Decision  Networks/Influence  Diagrams:  Yes  I  I  No  [3 

c.  Both  Bayesian  Nets  and  Influence  Diagrams:  Yes  l~~1  No  13 

2.  Model  Building  capability: 

a.  Using  GUI:  Yes  |3  No  □ 

b.  From  databases! i.e.  Structure  Learning):  Yes  l~~l  No  3 

c.  Other:  via  API  (i.e.  JAVA) 

3.  Type  of  graph  support: 

a.  Only  Undirected  graph:  Yes  I  I  No  13 

b.  Only  Directed  graph:  Yes  13  No  I  I 

c.  Chain  graphs  (i.e.  mixed  undirected  and  directed):  Yes  Q  No  3 

4.  Conditional  Probability  Table: 

a.  Specify  Tables:  Yes  £3  No  O 

b.  Specify  Equations:  Yes  I  I  No  13 

c.  From  databases  (learn  from  training  sets):  Yes  I  I  No  E3 

d.  Other: 

5.  Validity  of  the  model: 

a.  Consistent  Probability  check  of  variables  in  the  net:  Yes  3  No  Q 

b.  Cyclic  Dependency  structure  check  among  variables:  Yes  13  No  l~1 

6.  Inference  Algorithm  used:  junction  tree 

7.  Analysis  techniques  provided:  (1)  standard  conditioning  of  variables,  and  (2) 
computer  of  correlations  and  cross  entropy  between  pairs  of  variables. 
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8.  For  tools  that  support  Bayesian  Networks  (Ref.  Question  1)  are  there  built  in 
methods  to  trim  the  network  or  otherwise  limit  computational  complexity  with 
“relevance  reasoning”  or  any  other  method:  Yes  I  I  No  [>3 
a.  If  yes,  explain: 

Additional  Comments: 


Performance  and  Extensibility 

1 .  Language  used  to  develop: 

a.  GUI:  JAVA 

b.  Computational  Engine:  C++ 

2.  Is  source  code  available  to  the  end  user:  Yes  I  I  No  [><] 

3.  Is  the  API  available  to  the  end  user:  Yes  1^1  No  I  I 

4.  Availability  of  the  tool’s  performance  measurement:  Yes  Q  No  [X] 

a.  If  yes,  please  indicate  the  procedure  in  measuring  the  performance: 

5.  Maximum  Nodes  a  single  model  can  support:  no  built  in  constraint-  depends  on 
complexity  of  the  graph  (clique  size  limited  to  20) .  Several  hunderds  of  nodes 
run  smoothly.  I  expect  3000  to  10000  nodes  is  feasible  in  sparse  models,  but  I 
never  tested  this. 

6.  Error  recovery  support:  Yes  I  I  No  [xl 

a.  If  yes,  explain:  Java  error  messages 

Additional  Comments: 
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Interoperability 

1 .  Ability  to  interface  with  databases:  Yes  I  I  No  1E1 

a.  If  Yes, 

i.  Using  ODBC:  Yes  □  No  IE] 

ii.  Databases  it  can  interface  directly  with  (list  them): 

b.  Does  it  support  SQL  interface:  Yes  I  1  No  IE] 

c.  Can  the  models  be  saved  to  a  database:  Yes  I  I  No  IE1 

2.  Format  used  to  the  save  the  model:  text  file  (ASCII),  possibly  encrypted.  Other 
save  formats  are  not  supported. 

3.  Export  the  model  to  other  applications:  Yes  I  I  No  IEI 

a.  If  yes,  list  the  applications  the  model  can  be  exported  to: 

4.  Ability  to  export  results/graphs  generated  from  the  “what-if  ’  scenarios  or  the 
analysis:  Yes  ^  No  Q 

a.  If  Yes,  list  applications  that  it  can  be  exported  to:  gif,  jpeg 

5.  Tools  uses  open  standards:  Yes  [X]  No  Q 

a.  If  yes,  explain:  inference  is  standard  junction  tree  (with  the  complete- 
neighbour  heuristic  for  constructing  the  junction  tree) 

6.  Provide  group  collaboration  on  model  building  and  analysis:  Yes  I  I  No  IEI 
Additional  Comments: 

Additional  Notes:  an  unrestricted  GUI-version  can  be  downloaded  free  of 
charge  to  evaluate  BayesBuilder.  See  webpage  at  the  beginning  of  the 
questionnaire.  This  version  contains  a  tutorial  and  a  manual  of  the  tool. 
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Tool  Name:  BayesiaLab 
Website:  www.bayesia.com 
Company’s  Name:  Bayesia  S.A. 


General  Features 

1.  Cost:  For  about  10-60  licenses  or  is  their  a  group  license,  etc 
1  License:  3450  Euros 

5  Licenses:  10  350  Euros 
10  Licenses:  17  250  Euros 
50  Licenses:  43  125  Euros 

2.  Maturity  of  the  tool:  BayesiaLab  is  on  the  market  since  2002,  but  the  first 
version  of  the  tool  has  been  initially  realized  in  an  academic  research  center 
in  1998 

3.  Targeted  Industries:  Industrial  companies,  Consulting  companies,  Information 
Technology  companies,  Pharmaceuticals  companies,  Research  centers  and 
Universities 

4.  Major  clients:  EDF,  Saint  Gobain,  Cap  Gemini  Ernst&Young,  Sumitomo 
Pharmaceuticals,  Parks  Canada,  CNRS 

5.  Type  of  Operating  system  supported:  All  the  OS  having  a  Java  Virtual 
Machine 

6.  Recommended  Hardware: 

a.  Processing  power:  >=1  Ghz 

b.  RAM:  >=256 

c.  Other: 
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7.  Architecture  support: 

a.  Client  -  Server  Model:  Yes  I  1  No  [X] 

b.  Desktop:  Yes  ER1  No  1  I 

Input  Manipulation 

1.  Type  of  Network  support: 

a.  Bayesian  Networks:  Yes  [Xl  No  I  I 

b.  Decision  Networks/Influence  Diagrams:  Yes  ^  No  □ 

When  going  thru  the  tutorial,  I  did  not  see  any  discussion  on  whether 
BayesiaLab  supports  decision  node  type.  I  did  realize  that  it  supports  the 
concept  of  Cost.  Is  this  what  you  had  in  mind  when  you  answered  Yes  for  the 
above  question?  Can  BayesiaLab  support  decision  node  types? 

Indeed,  BayesiaLab  supports  two  kinds  of  costs:  a  cost  for  knowing  the  value  of  a 
variable  (useful  for  the  adaptive  questionnaires)  and  utility  nodes  that  allows 
qualifying  states  defined  by  node  value  combinations.  But,  we  have  not  implemented 
the  so  called  decision  nodes.  We  manage  such  nodes  with  probabilistic  nodes  (with 
action  as  modality)  to  evaluate  policies. 

c.  Both  Bayesian  Nets  and  Influence  Diagrams:  Yes  [x]  No  I  1 

2.  Model  Building  capability: 

a.  Using  GUI:  Yes  E<]  No  I  I 

b.  Other:  Using  a  custom  language,  batch  mode  etc 

c.  From  databases  (i.e.  Structure  learning):  Yes  [>3  No  Q 

3.  Type  of  graph  support: 

a.  Only  Undirected  graph:  Yes  I  I  No  Ex] 

b.  Only  Directed  graph:  Yes  I  I  No 

c.  Chain  graphs  (i.e.  mixed  undirected  and  directed):  Yes  No  O 

4.  Conditional  Probability  Table: 

a.  Specify  Tables:  Yes  [X]  No  Q 

b.  Specify  Equations:  Yes  153  No  I  I 

c.  From  databases  (learn  from  training  sets):  Yes  153  No  I~1 

d.  Other:  deterministic  mode 
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i.  Deterministic  mode  =>  if  a  node  is  logical  and,  or,  etc  deterministic 
mode  allows  for  this  logical  representation. 

Does  BayesiaLab  support  Noisy-or  nodes? 

It  is  possible  to  quickly  define  a  CPT  thanks  to  the  Noisy-Or-Function  that  is 
available  in  the  equation  editor  but  we  don’t  have  specific  Noisy-Or  nodes. 

5.  Validity  of  the  model: 

a.  Consistent  Probability  check  of  variables  in  the  net:  Yes  [X]  No  I  I 

b.  Cyclic  Dependency  structure  check  among  variables:  Yes  ^  No  Q 

6.  Inference  Algorithm  used:  Junction  tree,  Likelihood  weighting 

Does  BayesiaLab  support  both  causal  and  diagnostic  inference?  Also,  does 
the  user  have  a  choice  to  decide  which  type  of  inference  algorithm  (exact  or 
stochastic)  to  use? 

Yes,  you  can  do  inference  in  any  directions,  from  symptoms  to  causes,  from 
causes  to  symptoms  and  any  combination  of  these  inferences.  The  inference  menu 
allows  you  choosing  between  the  exact  and  approximate  inference. 

Does  BayesiaLab  support  negative  and  likelihood  evidence  (like  Hugin  and 
Netica) 

Yes,  for  both  exact  and  approximate  inference. 

7.  Analysis  techniques  provided: 

*  Arc  Analysis:  allows  highlighting  the  importance  of  the  arc  with  respect  to 
the  complete  structure.  The  thickness  of  the  arc  is  proportional  to  the 
importance  of  the  probabilistic  relation  it  represents  in  the  total  probability 
law. 

*  Target  Node  Analysis:  allows  the  visualization  of  the  quantity  of  information 
brought  by  each  node  for  the  knowledge  of  the  target  node.  The  brightness  of 
the  squares  appearing  inside  the  nodes  is  proportional  to  this  quantity  of 
information 

*  Target  Modality  Analysis:  allows  visualizing,  for  each  node,  two  kinds  of 
information  relative  to  the  target:  the  type  of  the  probabilistic  relation  binding 
this  variable  and  the  target  variable,  and  the  information  gain  brought  by  each 
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node  for  the  knowledge  of  the  modality  of  the  target  node 

*  Target  Analysis  Report:  Textual  report  describing  the  context  of  the  analysis 
(list  of  the  variables  that  are  observed  when  the  analysis  is  carried  out),  the 
probability  distribution  of  the  target  variable  knowing  the  context,  a  list  of 
nodes,  sorted  by  descending  order  according  to  their  relative  contribution  to 
the  knowing  of  the  target  variable  (the  nodes  that  do  not  bring  any 
information  do  not  appear  in  this  list),  for  each  value  of  the  target,  list  of 
nodes,  sorted  by  descending  order  according  to  their  relative  contribution  to 
the  knowing  of  the  target  value  (if  the  node  has  only  two  modalities,  this  list  is 
identical  to  the  preceding  one),  for  each  influencing  node,  description  the 
modal  value  with  respect  to  the  context  and  to  the  observed  modality  of  the 
target  node  (this  modal  value  comes  with  its  probability,  this  section  allows 
establishing  the  profile  of  this  target  value),  for  each  influencing  node, 
description  of  the  modal  value  when  the  target  node  is  unobserved  (but 
knowing  the  context,  that  makes  it  possible  to  define  the  profile  when  the 
target  variable  is  unobserved),  and  a  measure  indicating  the  variation  between 
the  modal  values  of  the  two  profiles.  This  value  is  measured  only  when  these 
values  are  identical.  A  positive  value  indicates  then  an  information  gain  to  the 
knowledge  of  the  node  value  when  the  target  value  is  observed. 

*  Adaptive  Questionnaires:  This  functionality  allows  the  automatic  and 
dynamic  activation  of  the  monitors  of  the  variables  that  bring  information  on 
the  probability  distribution  of  the  target  variable  at  lower  cost.  The  monitors 
are  sorted  according  to  their  relevance  (ratio  between  information  gain  in  the 
current  context  and  the  cost  implied  for  knowing  the  value  of  the  variable). 

*  Questionnaire  based  on  the  knowledge  of  a  particular  value  of  the  target 
variable:  This  functionality,  similar  to  the  previous  one,  differs  only  by  the 
measurement  of  the  information  gain.  This  gain  is  no  longer  measured 
according  to  the  total  probability  distribution  of  the  target  variable,  but  is 
completely  centered  on  the  knowledge  of  a  particular  value 

*  Causal  analysis  thanks  to  the  equivalence  classes.  Arcs  that  can  be  reversed 
without  changing  the  probability  law  are  undirected.  It  is  then  possible  to 
choose  an  orientation  that  is  automatically  propagated  to  the  network 


127 


(orientation  of  the  compelled  arcs,  update  of  the  conditional  probability 
tables). 

8.  For  tools  using  Bayesian  Networks  (Ref.  Col  B  data)  are  there  built  in  methods  to 
trim  the  network  or  otherwise  limit  computational  complexity  with  “relevance 
reasoning”  or  any  other  method:  Yes  I  I  No  I  I 

a.  If  yes,  explain:  I  don't  understand  if  you  are  speaking  about  building  the 
networks  through  data  mining  or  if  you  are  speaking  about  inference  in 
large  Bayesian  networks.  In  the  first  case,  our  learning  methods  are  all 
based  on  the  Minimum  Description  Length  score  (equivalent  to  having  an 
automatic  threshold  in  the  constraint  based  approaches  for  determining 
statistical  independence).  Concerning  inference,  it  is  indeed  impossible  to 
construct  the  junction  tree  for  large  networks  that  are  highly  connected.  In 
that  case,  on  has  to  use  the  approximate  inference. 

Thru  this  question,  we  were  trying  to  find  out  if  the  tool  has  the  capability  to  limit 
its  inference  based  on  the  type  of  evidence  provided.  For  example,  given  an  evidence 
for  a  certain  node,  the  tool  know  that  it  does  not  have  to  update  the  entire  net,  but 
only  update  a  sub-section  of  the  net  that  will  be  effected  by  this  evidence.  Does 
BayesiaLab  have  such  a  feature? 

Not  for  the  moment. 

Performance  and  Extensibility 

1 .  Language  used  to  develop: 

a.  GUI:  java 

b.  Computational  Engine:  java 

2.  Is  source  code  available  to  the  end  user:  Yes  I  I  No  1X1 

3.  Is  the  API  available  to  the  end  user:  Yes  [X]  No  I~1 
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4.  Availability  of  the  tool’s  performance  measurement:  Yes  ^  No  O 

Thru  this  question,  we  wanted  to  know  if  you  had  any  results  on  performance 
(time  it  took  to  do  inference)  of  the  tool  for  some  “typical  problems”.  Do  you 
have  any  results  that  show  the  performance  of  the  tool  for  any  problems  that  you 
have  modeled  and  used? 

Indeed  we  have  such  results  on  benchmarks  but  only  for  learning  (scientific 
publications  on  our  learning  algorithms),  but  not  for  inference. 

a.  If  yes,  please  indicate  the  procedure  in  measuring  the  performance:  The 

first  global  performance  measure  is  the  MDL  score  available  thanks 
to  the  console.  Otherwise,  BayesiaLab  offers  various  tools  for 
measuring  the  performance  when  specifying  a  target  node  : 

*  Total  precision  of  the  model:  number  of  correct  predictions  of  the 
value  of  the  target  variable  /  number  of  cases  in  the  data  base 

*  Confusion  matrix:  the  total  precision  is  useful  but  can  be  too 
general.  The  Confusion  Matrix  proposed  by  BayesiaLab  allows 
having  a  more  precise  feedback  about  the  model  performances: 
reliability  (i.e.  the  rate  of  correct  prediction  for  each  value  given  by 
the  model),  precision  (the  rate  of  detection  for  each  value  of  the  base) 

*  Lift  curve:  represents  the  detection  rate  of  the  target  value  (Y-axis) 
with  respect  to  the  number  of  processed  cases  (X-axis)  based  on  the 
order  defined  by  the  model.  Allows  choosing  the  best  threshold  to 
reach  a  particular  rate  of  detection 

*  ROC  curve:  True  Positive  Rate  (Y-axis)  against  the  False  Positive 
Rate  (X-axis).  Allows  choosing  the  best  threshold  to  reach  a  particular 
ratio  between  true  and  false  positives. 

5.  Maximum  Nodes  a  single  model  can  support:  As  I  said  it  before,  it  is  difficult  to 
give  a  maximum  number  of  nodes  since  it  depends  also  on  the  number  of 
node  values.  However,  last  week,  I  have  worked  on  a  genetic  data  base 
containing  2000  columns  with  3  modalities  for  each  gene  on  a  1  GHz  PC  with 
256  MO  without  problem. 
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6.  Error  recovery  support:  Yes  [x]  No  I  I 

a.  If  yes,  explain:  Exceptions  throwned  by  the  Java  Virtual  Machine  are 
catched  and  it  is  possible  to  automatically  send  the  error  message  to 
Bayesia  so  that  we  can  correct  the  error.  If  the  exception  is  relative  to 
memory,  a  specific  window  explain  what  parameters  have  to  be 
change  to  allow  BayesiaLab  to  work  with  more  memory. 


Interoperability 

1 .  Ability  to  interface  with  databases:  Yes  I  I  No  Ex] 

a.  If  Yes, 

i.  Using  ODBC:  Yes  |EI  No  □ 

1.  If  Yes,  list  the  data  bases  the  ODBC  driver  supports: 

ii.  Databases  it  can  interface  directly  with  (list  them): 

b.  Does  it  support  SQL  interface:  Yes  fX]  No  [  I 

c.  Can  the  models  be  saved  to  a  database:  Yes  Q  No  [X] 

2.  Format  used  to  the  save  the  model:  XBL,  a  specific  format  based  on  XML. 
BayesiaLab  can  also  read  BIF  and  NET  networks. 

3.  Export  the  model  to  other  applications:  Yes  I  I  No  [X] 

a.  If  yes,  list  the  applications  the  model  can  be  exported  to: 

4.  Ability  to  export  results/graphs  generated  from  the  “what-if  ’  scenarios  or  the 
analysis:  Yes  fX]  No  I  I 

a.  If  Yes,  list  applications  that  it  can  be  exported  to:  It  is  possible  to  paste 
the  monitors  to  any  applications  that  interact  with  the  clipboard  of 
the  OS.  Monitors  can  be  pasted  as  texts,  arrays  or  images. 
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5.  Tools  uses  open  standards:  Yes  [X]  No  Q 

a.  If  yes,  explain:  Bayesian  Interchange  Format  (BIF) 


6.  Provide  group  collaboration  on  model  building  and  analysis:  Yes  [X]  No  1  I 
Can  you  please  explain  how  group  collaboration  is  supported? 

We  have  two  tools  that  help  working  in  collaboration  on  model  building: 

1 .  It  is  possible  to  associate  color  tags  to  nodes  in  order  to  group  them  visually 
and  to  ease  the  brain  storming  with  the  domain  experts. 

2.  Its  is  possible  to  associate  hypertext  comments  (with  links  to  files  or  http 
address)  to  the  network  (by  default  the  Author  name  and  creation  date)  and  to 
each  node  (to  describe  the  role  of  the  node,  to  increase  the  traceability,  to 
justify  the  probabilities,  to  associate  illustrations,  schemas, ...) 


Additional  Notes:  BayesiaLab  also  offers: 

*  Complete  wizards  for  importing  data  bases  with  complete  missing  value 
processing  (filtering,  replacing  with  given  values,  structural  Expectation- 
Maximization),  filtering  capabilities  (variables  and  values),  intelligent 
discretization  tools,  database  transposition  (e.g.  for  the  micro-array  data 
processing). 

*  Possibility  to  generate  cases  with  respect  to  the  current  network 

*  Possibility  to  save  the  database  as  associated  to  the  current  network,  i.e.  with 
its  treatments  (discretization,  filtering,  missing  values  processing) 

*  Complete  structural  learning  toolbox:  three  unsupervised  learning  methods  to 
discover  all  the  probabilistic  relations  that  hold  in  the  database,  five  supervised 
learning  methods  to  characterize  a  target  variable  (from  naive  to  markov 
blanket  learning),  and  clustering  methods  that  allow  to  segment  the  data.  All 
these  learning  algorithms  support  missing  values. 

*  Hidden  variables:  it  is  possible  to  add  hidden  nodes,  i.e.  nodes  that  have  not 
data  in  the  database  and  to  combine  them  manually/automatically  with  the 


others. 


*  Possibility  to  do  structural  learning  from  an  existing  network  (a  priori 
knowledge),  and  to  fix  the  arcs  so  that  the  learning  algorithm  cannot  change  this 
knowledge. 

*  Batch  exploitation:  Possibility  to  use  the  current  network  to  do  an  off  line 
tagging  of  the  cases  contained  in  a  database. 

*  Complete  Dynamic  Bayesian  Networks  processing  to  introduce  the  temporal 
dimension  in  the  model  (step  by  step  simulation,  period  simulation,  probability 
evolution  in  a  graph  that  can  be  printed  or  saved,  and  possibility  to  associate  an 
observation  file).  Possibility  to  use  a  time  node,  i.e.  a  parameter  node  that 
represents  the  time  and  that  can  be  directly  used  in  the  equations  describing 
probability  distributions. 

*  Constraint  nodes:  Boolean  nodes  that  are  always  observed  and  that  are  used  to 
express  constraints  between  nodes 

*  Automatic  positioning  of  the  nodes 

*  Warning  tags  on  the  nodes  that  have  a  not  a  filled  conditional  probability  table 

*  Error  tags  on  the  nodes  that  have  an  incorrect  equation  due  to  parent 
modifications 

*  Hypertext  comments  on  the  networks  and  the  nodes 

*  Color  tags  on  the  nodes  to  group  them  semantically. 

*  Special  monitors  for  the  numerical  nodes  that  give  the  mean  and  the  standard 
deviation  and  that  have  three  display  modes  (normal,  relative  and  relative  + 
curve) 

*  Possibility  to  display  the  delta  (probability  variation)  on  the  monitor  between 
each  evidence  introduction/removal 

*  Possibility  to  specify  costs  to  each  node,  i.e.  the  cost  implied  for  knowing  the 
value  of  the  nodes  (useful  for  the  adaptive  questionnaires) 

*  Possibility  to  set  an  image  background  (a  map,  a  logo, ...) 


132 


Tool  Name:  OCCAM  -  Organizational  and  Cultural  Criteria  for 

Adversary  Modeling 

Website:  http://www.cra.com 

Company  Name:  Charles  River  Analytics,  Inc. 

General  Features 


General  Note: 

The  following  questionnaire  pertains  mainly  to  Bayesian  network  technology  which 
is  only  one  part  of  our  intelligence  operations  decision-aiding  system.  In  addition,  our 
system  allows  the  user  to  construct  and  fire  rules  using  a  commercial,  off-the-shelf  rule 
engine  and  editor.  Both  the  rules  and  belief  networks  are  able  to  interact  with  the  various 
entities  represented  in  our  tool  (e.g.,  cultural  stereotypes,  organizations,  individuals).  In 
addition,  our  primary  display,  the  social  network,  allows  the  user  to  quick  identify 
individuals,  groups,  and  events  (and  the  relationships  among  these)  to  run  these  rules  and 
belief  networks  on.  The  application  level  data  is  stored  in  a  database,  mapped  onto  a 
domain  model,  and  then  operated  on  by  rules  and/or  belief  networks  in  a  unified  fashion 
via  our  inference  engine  user  interface. 


1.  Ease  of  use: 

■  Is  there  feedback  for  processing  delays?  For  most  types  of  delays,  yes  (e.g.,  for  rule/BN  processing) 

■  Do  menu  items  and  text  descriptions  use  language  that  is  commonly  understood?  Yes 

■  Are  menu  items  in  standard  places?  Are  layouts,  terms,  icon  screen  positions,  windows,  dialog  boxes  consistent? 
Yes 

■  How  many  steps  does  it  take  to  perform  a  frequently  used  action  (desire  not  buried  in  menus;  not  multi-step)?  It 

depends  on  the  action  -  many  actions  are  tied  to  short-cut  keys.  Furthermore,  some  actions  are  tied  to 
multiple  user  interface  elements  (e.g.,  a  toolbar  button,  a  short-cut  key,  and  a  pull-down  menu  item)  to 
promote  both  ease-of-use  and  ease-of-learning. 

■  Do  you  show  user  that  he/she  has  made  a  mistake?  How  is  user  aided  in  correcting  mistakes?  It  depends  on  both 
where  in  the  system  the  mistake  is  made  and  what  type  of  mistake  is  made. 

■  Do  you  visually  display  system  objects,  actions,  and  options  to  users  or  are  these  available  textually?  How  do  you 
aid  users’  memory  for  these  choices?  Are  instructions  easily  available  if  the  user  forgets  or  loses  his/her  way? 

Both  toolbars  and  menus  are  used  to  present  available  actions,  objects,  preferences,  etc.  ToolTips  are 
available  for  all  toolbar  items. 

■  Does  your  system  cater  to  novice  and  expert  users?  Allow  users’  to  create  shortcuts  for  frequently  used  actions? 

The  system  does  cater  to  both  novice  and  expert  users  (in  particular,  by  providing  explanatory  interfaces 
and  visualizations  to  the  novice,  and  more  sophisticated  editing  and  deductive  reasoning  tools  to  the  expert). 
We  don’t  currently  allow  users  to  dynamically  generate  shortcuts,  but  we  provide  them  ourselves  (e.g.,  ctrl- 
a  selects  all,  ctrl-s  saves,  etc.) 

■  How  is  help  and/or  documentation  provided  to  the  user?  Tooltips  provide  additional  help  on  all  actions  and  in 
some  displays.  Help  menu  functionality  exists,  but  full  technical  documentation  is  still  underway. 

■  How  does  your  software  handle  making  best  use  of  limited  screen  “real  estate”?  Use  of  information  rich 
displays  coupled  with  extensive  use  of  scrollbars  and  split-screens  maximize  screen  “real  estate”. 

■  Top  5  “trouble”  reports?  N/A 
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2.  Cost:  Seat  license;  upgrades;  support;  maintenance;  (for  government  customer) 

Proprietary 

3.  Maturity  of  the  tool:  (Number  of  years  in  the  market,  #  versions/upgrades) 

Components  of  the  tool  have  been  under  development  for  5+  years,  other 
components  are  the  result  of  more  recent  (past  2  years)  research  and 
development  efforts. 

4.  Targeted  Industries: 

Proprietary 

5.  Major  clients: 

Proprietary 

6.  Type  of  Operating  system  supported:  List  all  applicable  OS 

WindowsXP  and  Windows2000,  currently,  plans  to  expand  to  Unix  platforms  in 
the  future. 

7.  Recommended  Hardware:  (list  both  minimal  and  optimal) 

a.  Processing  power:  P4  1.6+ 

b.  RAM:  512  MB 

c.  Other: 

8.  Architecture  support: 

a.  Client  -  Server  Model:  Yes  O  No 

b.  Desktop:  Yes  [Xl  No  I  I 
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Input  Manipulation 


1 .  Type  of  Network  support: 

a.  Bayesian  Networks:  Yes  Ex]  No  1  I 

b.  Decision  Networks/Influence  Diagrams:  Yes  I  I  No  Rl 

c.  Both  Bayesian  Nets  and  Influence  Diagrams:  Yes  Q  No  [x] 

d.  Decision  Trees:  Yes  O  No  ^ 

2.  Model  Building  capability: 

a.  Using  GUI:  Yes  |E1  No  □ 

b.  Other:  Using  a  custom  language,  batch  mode  etc 

c.  From  databases  (i.e.  Structure  learning):  Yes  I~1  No  [x] 

3.  Type  of  graph  support: 

a.  Only  Undirected  graph:  Yes  I  I  No  fXl 

b.  Only  Directed  graph:  Yes  fxl  No  I  I 

c.  Chain  graphs  (i.e.  mixed  undirected  and  directed):  Yes  I  I  No  1^1 

4.  Conditional  Probability  Table: 

a.  Specify  Tables:  Yes  ^  No  Q 

b.  Specify  Equations:  Yes  l~~l  No  153 

(not  currently  available  within  the  CPT  editor,  but  full  cut  and  paste  capability 
to/from  Excel  is  supported) 

c.  From  databases  fleam  from  training  sets):  Yes  [Xl  No  1  I  (supports  fully 
observable  and  partially  observable  data  sets) 

d.  Other:  Various  other  proprietary  user  interface  techniques. 

5.  Validity  of  the  model: 

a.  Consistent  Probability  check  of  variables  in  the  net:  Yes  [3  No  CD 

b.  Cyclic  Dependency  structure  check  among  variables:  Yes  [x]  No  O 

6.  Inference  Algorithm  used: 

Hugin  Junction  Tree  Algorithm  (not  product,  just  algorithm) 
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7.  Analysis  techniques  provided/supported:  List  and  explain 

Mutual  information  sensitivity  analysis 

8.  For  tools  using  Bayesian  Networks  (Ref.  Col  B  data)  are  there  built  in  methods  to 
trim  the  network  or  otherwise  limit  computational  complexity  with  “relevance 
reasoning”  or  any  other  method:  Yes  I  1  No  [x] 

a.  If  yes,  explain: 

Performance  and  Extensibility 

1 .  Language  used  to  develop: 

a.  GUI:  Java 

b.  Computational  Engine:  Java  (for  both  rules  and  BNs) 

2.  Is  source  code  generally  available  to  a  government  end  user:  Yes  Q  No  ^ 

(If  no,  is  there  any  circumstance  where  the  source  code  could  be  made  available?) 

This  would  have  to  be  negotiated. 

3.  Is  the  API  available  to  the  end  user:  Yes  [Xl  No  I  I 

4.  Availability  of  the  tool’s  performance  measurement:  Yes  n  No  [>3 

a.  If  yes,  please  indicate  the  benchmark/procedure  used  to  measure  performance: 

5.  Maximum  Nodes  a  single  model  can  support:  (Max  nodes  a  model  can  support  when 
running  on  the  recommended  system) 

Only  limited  by  CPU  and  memory.  On  the  recommended  system,  on  the  order  of 
103  (but  still  depends  on  network  density). 

6.  Error  recovery  support  (if  something  crashes,  what  happens?): 

Use  last  saved  state  (no  intermediate  or  auto-save  features  currently  implemented) 
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Interoperability 


1 .  Ability  to  interface  with  databases:  Yes  ^  No  O  (The  application  supports  SQL 
queries  to  retrieve  data  used  as  evidence  in  the  networks.  Also,  the  rule  engine 
component  can  store  rules  in  a  database.  Bayesian  network  component  receives 
data  from  the  database  via  the  domain  model  of  the  main  application.) 

a.  If  Yes, 


i.  Using  Open  Data  Base  Connectivity  (ODBC):  Yes  1^1  No  I  I 

1 .  If  Yes,  list  the  data  bases  the  ODBC  driver  supports:  the  JDBC 
2.0  bridge  used  supports  the  ODBC  2.x  and  ODBC  3.x 
driver  manager  and  drivers,  so  any  database  that  supports 
those  standards 

ii.  Databases  it  can  interface  directly  with(list  them):  anything 
compatible  with  the  JDBC/ODBC  bridge 

b.  Does  it  support  SQL  interface:  Yes  [X]  No  O 

c.  Can  models  created  in  the  tool  be  saved  to  a  database:  Yes  [X]  No  I  I 

(not  Bayesian  networks,  the  application  level  data  is  saved  in  a  database 
and  any  ruleset  may  also  be  stored  in  a  database) 


2.  Format  used  to  the  save  the  model:  xbn  (standard  xml  BN  format)  for  BNs 
proprietaiy  ruleset  format  for  rules 


3.  Export  the  model  to  other  applications:  Yes  [x]  No  I  I 

a.  If  yes,  list  the  applications  the  model  can  be  exported  to: 

Any  application  that  will  read  the  common  xbn  format. 
Rulesets  are  not  currently  exportable 


4.  Ability  to  export  results/graphs  generated  from  the  “what-if  ’  scenarios  or  the 
analysis:  Yes  1^1  No  I  I 

a.  If  Yes,  list  applications  that  it  can  be  exported  to: 

This  feature  is  currently  under  development: 

BNs:  the  model  can  be  exported  to  any  application  that  reads  xbn  files 
Rules:  the  model  cannot  currently  be  exported 
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Domain  model:  the  model  is  stored  in  a  database  accessible  via  SQL  queries 
Also,  results  of  analysis  can  be  captured  in  standard  image  formats  (e.g., 
JPEG,  GIF) 

5.  Tools  uses  open  standards:  Yes  [X]  No  [  I 

a.  If  yes,  explain:  In  the  BN  model:  save  model  to  xbn  files  (xml  bnet 

stadard),  cut/paste  CPT  entries  to/from  Excel.  In  the  domain  model:  SQL 
syntax  for  querying  data  in  domain  model 

6.  Provide  group  collaboration  on  model  building  and  analysis:  Yes  Q  No  ^ 
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Tool  Name:  DXpress  Solution  Series 
Website:  www.kic.com 

Company’s  Name:  Knowledge  Industries,  Inc. 

General  Features 

1 .  Cost:  To  be  negotiated  -  depends  upon  user  domain,  degree  of  exclusivity  and 
usage.  Some  domains  are  not  available  for  license. 

2.  Maturity  of  the  tool:  Fully  mature  software  developed  since  1992,  no  changes 
to  the  Inference  Engine  since  1999  and  only  minor  editing  for  the  editor  and 
testing  segments.  No  outstanding  trouble  reports,  in  daily  heavy  commercial 
usage  for  web-based  interactive  diagnostic  applications.  First  sales  in  1993. 

3.  Targeted  Industries:  The  initial  version  was  designed  for  medical  diagnosis. 
The  current  software  has  been  developed  to  be  domain  independent  and  has 
been  used  in  aerospace,  metals  manufacturing,  medical  clinics,  locomotive 
repair,  automotive  repair,  internal  auditing  and  community  relations  as  well 
as  proprietary  applications.. 

4.  Major  clients:  The  Knowledge  Industries'  client  list  is  proprietary  and  will  be 
disclosed  as  appropriate.  It  includes  Fortune  100  firms  as  well  as  start-up 
companies. 

5.  Type  of  Operating  system  supported:  Microsoft  W2000,  NT4.0,  XP 

6.  Recommended  Hardware: 

a.  Processing  power:  500  mhz  or  greater 

b.  RAM:  Inference  Engine  - 1.4  MB,  Compilier  -  5.7  MB,  Test  System  - 
2.5  MB 

c.  Other:  Sufficient  disk  space  to  store  the  Bayesian  Knowledge 
Databases.  Each  Dayes  network  is  compiled  by  DXpress  and  is  stored 
as  an  Object  (file)  that  can  be  executed  by  either  WIN-DX  for  test  and 
validation  purposes  or  operated  by  the  API/dll  that  resides  within  the 
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End  User’s  application.  We  refer  to  these  in  the  collective  as  the 
Knowledge  Databases. 

Can  you  please  explain  what  you  mean  by  Bayesian  Knowledge 
Databases? 

7.  Architecture  support: 

a.  Client  -  Server  Model:  Yes  1X1  No  I  I 

b.  Desktop:  Yes  1X1  No  I  I 

Additional  Comments:  There  are  three  separate  modules.  DXpress  is  the 
Editor/Compilier,  Win-DX  is  the  test  and  validation  module  and  API/dll  is  the 
Inference  Engine  that  executes  the  compiled  Bayes  network.  Standard  practice  is  to 
develop  and  test  on  the  desktop  and  upload  to  the  Inference  Engine  running  on  a 
server.  The  Inference  Engine  (API/dll)  is  state-free. 


Input  Manipulation 

1.  Type  of  Network  support: 

a.  Bayesian  Networks:  Yes  [XI  No  I  I 

b.  Decision  Networks/Influence  Diagrams:  Yes  I  I  No  I  I 

c.  Both  Bayesian  Nets  and  Influence  Diagrams:  Yes  I  I  No  I  1 

2.  Model  Building  capability: 

a.  Using  GUI:  Yes  No  □ 

b.  From  databases(i.e.  Structure  Learning):  Yes  [3  No  3 

c.  Other:  There  is  a  graphics  editor  for  building  the  Bayes  Network. 

3.  Type  of  graph  support: 

a.  Only  Undirected  graph:  Yes  I  I  No  I  I 

b.  Only  Directed  graph:  Yes  3  No  O 

c.  Chain  graphs  (i.c.  mixed  undirected  and  directed):  Ye3  [3  No  I  I 
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4.  Conditional  Probability  Table: 

a.  Specify  Tables:  Yes  fX]  No  I  I 

b.  Specify  Equations:  Yes  I  I  No  1^1 

c.  From  databases  (learn  from  training  sets):  Yes  I~1  No 

d.  Other:  There  currently  is  no  direct  method  of  connecting  external  sources  of 
Conditional  Probabilities  to  the  CP  tables.  Such  a  feature  has  been  designed 
and  can  be  added  if  needed.  All  CP  tables  entries  must  be  completed  prior  to 
compilation.  Incomplete  information  is  not  allowed. 

Is  there  support  for  Noisy-Or  nodes  (if  a  child  node  has  a  large  parent  set)? 

Yes 

Does  DXpress  support  negative  and  likelihood  evidence?  (Like  Hugin  and 
Netica) 

I’m  sorry;  I  have  not  examined  either  of  these  packages  in  enough  detail  to 
respond.  I  do  not  know  what  is  meant  by  negative  and  likelihood  evidence  as 
opposed  to  Conditional  Probabilities.  I  suspect  that  this  is  a  result  of 
differences  in  nomenclature,  not  capability. 

5.  Validity  of  the  model: 

a.  Consistent  Probability  check  of  variables  in  the  net:  Yes  ^  No  O 

b.  Cyclic  Dependency  structure  check  among  variables:  Yes  E><3  No  I~1 

6.  Inference  Algorithm  used:  Proprietary,  to  be  discussed  under  conditions  of 
confidentiality  as  needed. 

Is  it  based  on  any  standard  junction  tree  algorithms  that  are  generally  applied  to 
Bayesian  Nets?  Also,  does  DXpress  support  both  exact  and  approximate  inference 
algorithms? 

A  response  to  this  question  will  be  forthcoming. 

7.  Analysis  techniques  provided:  Execution  of  the  network  may  be  coded  directly 
into  the  system  so  that,  depending  upon  the  state  of  an  input  variable,  entire 
segments  of  the  network  will  be  deemed  "not  applicable"  and  will  not  be 
examined  -  i.e.,  yes. 
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For  this  question,  I  was  thinking  along  the  lines  of  Sensitivity  analysis  or 
Importance  analysis  -  where  the  user  selects  a  node  (target  node)  and  wants  to 
find  out  how  the  other  nodes  in  the  network  impact  it.  Does  DXpress  provide  such 
support? 

Ah  ha!  Yes,  we  have  what  we  call  a  “debug”  function  that  allows  the  elicitor 
to  identify  selected  nodes  to  be  “debug”  nodes  and  a  complete  sensitivity 
analysis  can  be  made  during  the  test  and  validation  phase  of  BBN 
construction  using  WIN-DX. 

Where  is  this  debug  function?  Also,  when  you  mean  by  sensitivity  analysis  during 
using  WIN-DX  ,  are  you  referring  to  how  the  beliefs  in  the  possible  disorders 
changes  based  on  the  evidence  entered. 

8.  For  tools  that  support  Bayesian  Networks  (Ref.  Question  1)  are  there  built  in 
methods  to  trim  the  network  or  otherwise  limit  computational  complexity  with 
“relevance  reasoning”  or  any  other  method:  Yes  1^1  No  I  I 
a.  If  yes,  explain:  The  technology  is  proprietary. 

Additional  Comments:  Value  of  Information  is  used. 

Performance  and  Extensibility 


1 .  Language  used  to  develop: 
a.  GUI:  C++ 


b.  Computational  Engine:  C++ 

2.  Is  source  code  available  to  the  end  user:  Yes  I  I  No  [x] 

3.  Is  the  API  available  to  the  end  user:  Yes  fXl  No  I  I 

4.  Availability  of  the  tool’s  performance  measurement:  Yes  1X1  No  I  I 
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a.  If  yes,  please  indicate  the  procedure  in  measuring  the  performance:  Load 
testing. 

I  am  not  clear  on  what  you  mean  by  Load  testing?  For  this  question,  we  want  to 
know  if  you  have  done  any  performance  measurements  of  DXpress  for  some  set 
of  problems.  If  so,  are  the  results  available? 

A  licensee  performed  load  tests  prior  to  selecting  our  software.  In  Summary 
form  -  on  a  455  MHz  computer  using  a  large  Bayes  Network  (over  200 
nodes)  and  instantiated  with  all  input  simultaneously  the  test  system 
processed  over  1,000  transactions  per  minute  with  no  measurable  delay. 

5.  Maximum  Nodes  a  single  model  can  support:  Performance  limit  has  not  been 
established,  however,  400-node,  multi-fault  networks  have  been  built  and 
operated.  In  general,  the  limitation  on  node  number  is  the  ability  of  the 
developer(s)  to  understand  the  resulting  network  at  a  deep  level.  We  have 
found  that  for  a  team  of  experienced  elicitors,  an  effective  limit  of  400  nodes 
per  network  applies.  We  solve  the  large  network  problem  by  a  domain 
division  method.  For  example,  in  one  major  application  of  over  700 
networks,  the  applicable  network  is  selected  by  a  meta-network. 

6.  Error  recovery  support:  Yes  fxl  No  I  I 

a.  If  yes,  explain:  Not  a  clear  question  in  this  context.  Each  module  has 
operational  diagnostics.  Do  you  refer  to  the  Inference  Engine?  The 
Editor/Compilier? 

We  want  to  find  out  what  kind  of  error  recovery  support  the  tools  provides  to  the 
user  when  developing  the  model  and  running  inference  on  it.  Say,  if  I  am 
constructing  a  model,  does  the  tool  have  a  built  in  feature  to  auto-save  the  model 
every  x  units  of  time,  so  if  my  system  crashes,  there  is  a  way  to  recover  the  last 
saved  version? 

We  do  not  have  automatic  “save”  implemented.  The  User  must  save  the 
BBN  as  it  is  developed. 
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For  API,  what  kind  of  Error  recovery  support  is  provided?  -  are  the  error 
messages  well  documented? 

We  think  the  error  messages  from  the  API  are  well  documented  -  but  none 
of  our  clients  have  reported  any  error  messages  in  the  last  four  years  of 
heavy  commercial  usage  of  the  API. 

When  running  the  inference  algorithm,  does  the  tool  let  the  user  know  if  it  ran  out 
of  memory? 

We  have  never  had  this  problem.  Our  inference  system  depends  upon  a  pre¬ 
compiled  BBN  in  contrast  to  interpretive  systems,  the  API  is  optimized  for  a 
small  footprint  and  even  the  largest  of  the  BBNs  that  we  have  tested  (over 
400  nodes)  will  run  on  a  Pentium  I  computer  with  limited  memory. 

Additional  Comments: 


Interoperability 

1 .  Ability  to  interface  with  databases:  Yes  fx]  No  I  I 

a.  If  Yes, 

i.  Using  ODBC:  Yes  □  No  □ 

ii.  Databases  it  can  interface  directly  with  (list  them):  The  potential 
exists,  but  has  not  been  implimented. 

I  have  noticed  that  DXpress  does  not  support  structure  learning  (true)  and 
conditional  probability  learning  (true).  So,  if  this  feature  is  implemented,  will  it 
useful  only  to  save  the  Bayesian  net  models  to  databases  directly  (we  do  save  our 
compiled  networks  and  they  could  easily  be  placed  into  a  database  system). 

No,  we  would  allow  the  importation  of  Conditional  Probability  tables  from 
external  sources  to  modify  an  existing  BBN. 

b.  Does  it  support  SQL  interface:  Yes  I  1  No  15<] . 

c.  Can  the  models  be  saved  to  a  database:  Yes  1^1  No  I  I 

2.  Format  used  to  the  save  the  model:  Proprietary. 


144 


3.  Export  the  model  to  other  applications:  Yes  I  I  No  1^1 

a.  If  yes,  list  the  applications  the  model  can  be  exported  to:  All  file  formats 
are  proprietary. 

4.  Ability  to  export  results/graphs  generated  from  the  “what-if  ’  scenarios  or  the 
analysis:  Yes  [X]  No  I  | 

a.  If  Yes,  list  applications  that  it  can  be  exported  to:  Any 
Can  you  please  list  some  of  the  applications? 

We  routinely  export  files  from  the  Test  and  Validation  module  (WIN-DX)  to 
Excel  and  other  similar  programs  in  a  comma-delimited  format  to  allow 
graphic  display  of  the  individual  Case  that  is  being  evaluated. 

5.  Tools  uses  open  standards:  Yes  I  |  No  fXl 

a.  If  yes,  explain: 

6.  Provide  group  collaboration  on  model  building  and  analysis:  Yes  E><^  No  I  | 

Additional  Comments:  Has  had  extensive  use  by  a  team  building  over  700 
networks  for  automotive  diagnosis. 


Additional  Notes:  Software  is  industrial-strength,  runs  all  of  the  time  and  runs 
well,  handles  both  single  fault  and  multiple  fault  Bayes  networks  and  is  feature 
rich.  Well  suited  for  human  elicitation  and  can  be  readily  adapted  for 
automation  of  conditional  probabilities  from  external  sources.  See 
www.Symptomedix.com  for  an  interactive  diagnostic  for  headaches.  Note  that 
the  sequence  of  the  questions  in  the  demonstration  is  based  upon  all  prior  inputs 
and  that  the  diagnosis  becomes  stable  after  the  first  8  to  12  of  the  150+  inputs 
have  been  requested. 
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Tool  Name:  Ergo 

Website:  www.noeticsystems.com/ergo 
Company’s  Name:  Noetic  Systems  Incorporated 

General  Features 

1.  Cost:  $200  GUI,  $100  run-time  library;  volume  discounts  are  available 

2.  Maturity  of  the  tool:  Developed  and  maintained  since  1989;  no  bug  reports  for 
version  1.0  (GUI  or  API)  for  over  1  year. 

3.  Targeted  Industries:  Experts  in  any  field  who  want  to  transfer  their  expertise 
to  a  computer-based  expert  system 

4.  Major  clients:  Licensees  include  Cisco  Learning  Institute,  General  Electric, 
Educational  Testing  Service,  dozens  of  university/academic  licensees  in 
North  America,  Asia,  and  Europe 

5.  Type  of  Operating  system  supported:  GUI:  Macintosh,  Windows. 

Run-time  API:  Macintosh,  Windows,  Unix 

6.  Recommended  Hardware: 

a.  Processing  power:  No  minimum 

b.  RAM:  1Mb 

c.  Other:  Disk  space:  1  Mb 

7.  Architecture  support: 

a.  Client  -  Server  Model:  Yes  I  I  No  [X] 

b.  Desktop:  Yes  [Xl  No  I  I 

Additional  Comments: 
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Input  Manipulation 


1 .  Type  of  Network  support: 

a.  Bayesian  Networks:  Yes  [Xl  No  I  I 

b.  Decision  Networks/Influence  Diagrams:  Yes  O  No  [X] 

c.  Both  Bayesian  Nets  and  Influence  Diagrams:  Yes  I  I  No  [xl 

2.  Model  Building  capability: 

a.  Using  GUI:  Yes  1^1  No  f~1 

b.  From  databases(i.e.  Structure  Learning):  Yes  [x]  No  I  I 

c.  Other:  From  scripts  (i.e.,  replay  network-construction  events) 

How  can  it  do  structure  learning  when  there  is  no  support  for  database  interface? 
Replay  network-construction  events???  Explain 

3.  Type  of  graph  support: 

a.  Only  Undirected  graph:  Yes  I  I  No  I  I 

b.  Only  Directed  graph:  Yes  [>3  No  Q 

c.  Chain  graphs  (i.e.  mixed  undirected  and  directed):  Yes  I  I  No  I  I 

4.  Conditional  Probability  Table: 

a.  Specify  Tables:  Yes  ^  No  Q 

b.  Specify  Equations:  Yes  I  I  No  153 

c.  From  databases  (learn  from  training  sets):  Yes  fXl  No  1  I 

d.  Other: 

5.  Validity  of  the  model: 

a.  Consistent  Probability  check  of  variables  in  the  net:  Yes  ^  No  Q 

b.  Cyclic  Dependency  structure  check  among  variables:  Yes  £3  No  Q 

6.  Inference  Algorithm  used:  Proprietary  version  of  Lauritzen-Spiegelhalter; 
user  can  also  enter  likelihood  ratios  as  evidence;  inference  can  be  performed 
in  batch  mode  for  validation,  from  GUI  or  API 

7.  Analysis  techniques  provided:  Batch  processing;  log  file 
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8.  For  tools  that  support  Bayesian  Networks  (Ref.  Question  1)  are  there  built  in 
methods  to  trim  the  network  or  otherwise  limit  computational  complexity  with 
“relevance  reasoning”  or  any  other  method:  Yes  I  I  No  1^1 
a.  If  yes,  explain: 

Additional  Comments:  Client-server  architecture  and  database  connectivity  are 
under  development. 


Performance  and  Extensibility 

1 .  Language  used  to  develop: 

a.  GUI:  C++ 

b.  Computational  Engine:  C++;  API  has  C  or  C++  interface 

2.  Is  source  code  available  to  the  end  user:  Yes  I  I  No  1^1 

3.  Is  the  API  available  to  the  end  user:  Yes  ^  No  O 

4.  Availability  of  the  tool’s  performance  measurement:  Yes  I  I  No  I  I 

a.  If  yes,  please  indicate  the  procedure  in  measuring  the  performance: 

5.  Maximum  Nodes  a  single  model  can  support:  Limited  by  available  memory; 
maximum  number  of  conditional  probabilities  per  node  is  2A32 

6.  Error  recovery  support:  Yes  1^1  No  I  I 

a.  If  yes,  explain:  GUI  and  API  return  error  codes  for  all 
errors/warnings 

Additional  Comments: 
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Interoperability 

1 .  Ability  to  interface  with  databases:  Yes  I  ]  No  Ex] 

a.  If  Yes, 

i.  Using  ODBC:  Yes  □  No  □ 

ii.  Databases  it  can  interface  directly  with  (list  them): 

b.  Does  it  support  SQL  interface:  Yes  I  I  No  Ex] 

c.  Can  the  models  be  saved  to  a  database:  Yes  Q  No  ^ 

2.  Format  used  to  the  save  the  model:  Proprietary  binary  (more  efficient);  text; 
XML 

3.  Export  the  model  to  other  applications:  Yes  CxH  No  1  I 

a.  If  yes,  list  the  applications  the  model  can  be  exported  to: 

4.  Ability  to  export  results/graphs  generated  from  the  “what-if  ’  scenarios  or  the 
analysis:  Yes  [X]  No  I  I 

a.  If  Yes,  list  applications  that  it  can  be  exported  to:  GUI  saves  graph  to 
clipboard  for  export  to  drawing  programs 

5.  Tools  uses  open  standards:  Yes  I  I  No  1X1 

a.  If  yes,  explain: 

6.  Provide  group  collaboration  on  model  building  and  analysis:  Yes  I  I  No  1^1 
Additional  Comments: 

Additional  Notes:  Inference  based  on  docking  has  also  been  implemented.  This 
advance  supports  the  decomposition  of  a  network  into  diagnostic  (system)  and 
evidence  subnetworks;  evidence  subnetworks  are  "docked"  to  the  diagnostic 
module  when  they  are  instantiated,  leading  to  greatly  reduced  computational 
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complexity  for  the  diagnostic  module,  without  loss  of  accuracy  as  occurs  for 
stochastic-inference  methods.  Please  see  the  following  reference  for  more 
information: 

Almond  RG,  Herskovits  EH,  Mislevy  RJ,  Steinberg  LS.  Transfer  of  information 
between  system  and  evidence  models.  Heckerman  D  and  Whittaker  J  (Eds.). 
Proceedings  of  the  Seventh  International  Workshop  on  Artificial  Intelligence 
and  Statistics,  1999,  pp.  181-186.  San  Francisco:  Morgan  Kaufmann. 
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Tool  Name:  Hugin  Developer 
Website:  www.hugin.com 
Company’s  Name:  Hugin  Expert  A/S 

General  Features 

1 .  Cost:  Please  see  price  quote  separately 

2.  Maturity  of  the  tool:  Hugin  API  -  13  years  -  Hugin  GUI  -  10  years 

3.  Targeted  Industries:  Medical  industy  -  this  was  the  starting  industry  -  now  as 
Hugin  is  a  general  purpose  tool,  we  aim  at  many  different  industries. 

4.  Major  clients:  Please  see  our  reference  list.  Note:  No  reference  list  provided. 

5.  Type  of  Operating  system  supported:  Windows,  Unix,  Linux 

6.  Recommended  Hardware: 

a.  Processing  power:  1  GHz 

b.  RAM:  256  Mb 

c.  Other: 

7.  Architecture  support: 

a.  Client  -  Server  Model:  Yes  I  I  No  1^1 

b.  Desktop:  Yes  [Xl  No  I  I 


Additional  Comments:  Hugin  API  compiles  with  ansi  C,  and  the  Hugin  GUI 
requires  Java.  It  is  Hugins  policy  to  be  open  for  the  requirements  of  the  customers. 
If  a  customers  e.g  uses  another  operating  system  that  the  mentioned  ones,  we  are 
open  for  this.  It  only  requires  that  the  customer  makes  the  operating  system 
available  to  Hugin  for  compiling. 
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Input  Manipulation 


1.  Type  of  Network  support: 

a.  Bayesian  Networks:  Yes  [><]  No  O 

b.  Decision  Networks/Influence  Diagrams:  Yes  ^  No  Q 

c.  Both  Bayesian  Nets  and  Influence  Diagrams:  Yes  ^  No  Q 

2.  Model  Building  capability: 

a.  Using  GUI:  Yes  K1  No  □ 

b.  From  databases(i.e.  Structure  Learning):  Yes  Ex]  No  I  I 

c.  Other: 

3.  Type  of  graph  support: 

a.  Only  Undirected  graph:  Yes  £x]  No  O 

b.  Only  Directed  graph:  Yes  ^  No  O 

c.  Chain  graphs  (i.e.  mixed  undirected  and  directed):  Yes  E*E1  No  I  I 

4.  Conditional  Probability  Table: 

a.  Specify  Tables:  Yes  EXl  No  I  1 

b.  Specify  Equations:  Yes  EX]  No  I  I 

c.  From  databases  (learn  from  training  sets):  Yes  [X]  No  CD 

d.  Other:  Supports  missing  information 

5.  Validity  of  the  model: 

a.  Consistent  Probability  check  of  variables  in  the  net:  Yes  EXl  No  I  I 

b.  Cyclic  Dependency  structure  check  among  variables:  Yes  No  I  I 

6.  Inference  Algorithm  used:  Hugin  Propagation  -  a.k.a  junction  tree 
propagation 

7.  Analysis  techniques  provided:  Conflict  analysis 
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8.  For  tools  that  support  Bayesian  Networks  (Ref.  Question  1)  are  there  built  in 
methods  to  trim  the  network  or  otherwise  limit  computational  complexity  with 
“relevance  reasoning”  or  any  other  method:  Yes  I  I  No  [Xl 
a.  If  yes,  explain: 

Additional  Comments:  3.a  -  easily  corrupted 

Performance  and  Extensibility 


1 .  Language  used  to  develop: 

a.  GUI:  JAVA 

b.  Computational  Engine:  C 

2.  Is  source  code  available  to  the  end  user:  Yes  I  1  No  fx] 

3.  Is  the  API  available  to  the  end  user:  Yes  [>3  No  O 

4.  Availability  of  the  tool’s  performance  measurement:  Yes  No  1  I 

a.  If  yes,  please  indicate  the  procedure  in  measuring  the  performance: 

Please  see  page  8,  9  and  16  in  the  attached  slides.  Note:  No  slides 
provided. 

5.  Maximum  Nodes  a  single  model  can  support:  RAM  limited  only,  please 
see  page  8  and  9  in  the  attached  slides.  Note:  No  slides  provided. 

6.  Error  recovery  support:  Yes  Cx]  No  I  I 

a.  If  yes,  explain:  All  functions  set  error  code 


Additional  Comments:  C++, VB, JAVA:  Throw/Catch  Mechanism 
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Interoperability' 


1.  Ability  to  interface  with  databases:  Yes  1^1  No  I  I 

a.  If  Yes, 

i.  Using  ODBC:  Yes  |E1  No  □ 

ii.  Databases  it  can  interface  directly  with  (list  them):  Oracle  8  i 

b.  Does  it  support  SQL  interface:  Yes  I  I  No  153 

c.  Can  the  models  be  saved  to  a  database:  Yes  I  I  No 

2.  Format  used  to  the  save  the  model:  NET,  HKB 

3.  Export  the  model  to  other  applications:  Yes  [x]  No  I  I 

a.  If  yes,  list  the  applications  the  model  can  be  exported  to:  B-course,  Genie, 
Samiam 

4.  Ability  to  export  results/graphs  generated  from  the  “what-if”  scenarios  or  the 
analysis:  Yes  1X1  No  I  I 

a.  If  Yes,  list  applications  that  it  can  be  exported  to:  BMP 

5.  Tools  uses  open  standards:  Yes  I  I  No  I  I 

a.  If  yes,  explain:  All  major  Algorithms  published,  ANSI  C  (API) 

6.  Provide  group  collaboration  on  model  building  and  analysis:  Yes  Q  No  1^1 
Additional  Comments: 

Additional  Notes;  In  general  Hugin  are  open  towards  special  customer  needs 
and  wishes.  This  means  that  if  our  customers  needs  any  further  functionality 
which  not  are  implemented  in  the  standard  product,  we  are  willing  to  consider 
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er  consultancy  to  the  customer  in 
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Tool  Name:  PSYOP  PT 

Website:  www.mtisurv.com/psyopdemo 

Company  Name:  Metrica,  Inc. 

General  Features 


1 .  Ease  of  use: 

■  Is  there  feedback  for  processing  delays?  Yes 

■  Do  menu  items  and  text  descriptions  use  language  that  is  commonly  understood?  Yes 

■  Are  menu  items  in  standard  places?  Are  layouts,  terms,  icon  screen  positions,  windows,  dialog  boxes 
consistent?  Yes 

■  How  many  steps  does  it  take  to  perform  a  frequently  used  action  (desire  not  buried  in  menus;  not 
multi-step)?  1  or  2  steps 

■  Do  you  show  user  that  he/she  has  made  a  mistake?  How  is  user  aided  in  correcting  mistakes? 

■  Do  you  visually  display  system  objects,  actions,  and  options  to  users  or  are  these  available  textually? 
How  do  you  aid  users’  memory  for  these  choices?  Are  instructions  easily  available  if  the  user  forgets 
or  loses  his/her  way?  Yes 

■  Does  your  system  cater  to  novice  and  expert  users?  Allow  users’  to  create  shortcuts  for  frequently  used 
actions?  Mainly  to  novice  users. 

■  How  is  help  and/or  documentation  provided  to  the  user? 

■  How  does  your  software  handle  making  best  use  of  limited  screen  “real  estate”?  Scrolling  and  user 
options  to  reduce  font  size. 

■  Top  5  “trouble”  reports? 


2.  Cost:  Seat  license;  upgrades;  support;  maintenance;  (for  government  customer) 

3.  Maturity  of  the  tool:  Number  of  years  in  the  market,  #  versions/upgrades. 

4.  Targeted  Industries:  Business,  medical,  govt.,etc.  US  Armed  Forces 

5.  Major  clients:  US  Air  Force  Air  Intelligence  Agency 

6.  Type  of  Operating  system  supported:  List  all  applicable  OS 

Server:  Windows  NT  4.0  or  higher;  Client:  Any  machine  capable  of  running 
Internet  Explorer  5.0  or  higher. 
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7.  Recommended  Hardware:  (list  both  minimal  and  optimal) 

a.  Processing  power:  200Mhz  or  Higher 

b.  RAM:  24  Mb  (  dependant  on  OS  ) 

c.  Other:  Server  software  required  IIS  4.0  or  higher 

8.  Architecture  support: 

a.  Client  -  Server  Model:  Yes  O  No  O  Yes. 

b.  Desktop:  Yes  Q  No  O  Yes,  if  the  desktop  has  Microsoft  PWS  or  IIS 
installed. 


Input  Manipulation 

1.  Type  of  Network  support: 

a.  Bayesian  Networks:  Yes  I  I  No  I  I 

b.  Decision  Networks/Influence  Diagrams:  Yes  I  I  No  I  I 

c.  Both  Bayesian  Nets  and  Influence  Diagrams:  Yes  CD  No  Q 

d.  Decision  Trees:  Yes  Q  No  d| 

2.  Model  Building  capability: 

a.  Using  GUI:  Yes  □  No  □  Yes 

b.  Other:  Using  a  custom  language,  batch  mode  etc 

c.  From  databases  (i.e.  Structure  learning):  Yes  |  |  No  |  1 

3.  Type  of  graph  support: 

a.  Only  Undirected  graph:  Yes  I  I  No  I  I 

b.  Only  Directed  graph:  Yes  □  No  □ 

c.  Chain  graphs  (i.e.  mixed  undirected  and  directed):  Yes  HU  No  O 

4.  Conditional  Probability  Table: 

a.  Specify  Tables:  Yes  I  I  No  I  I 

b.  Specify  Equations:  Yes  I  I  No  I  I 

c.  From  databases  (learn  from  training  sets):  Yes  D  No  □ 

d.  Other:  Other  Input  techniques  used 
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5 .  Validity  of  the  model : 

a.  Consistent  Probability  check  of  variables  in  the  net:  Yes  □  No  □ 

b.  Cyclic  Dependency  structure  check  among  variables:  Yes  □  No  □ 

6.  Inference  Algorithm  used:  List  all  relevant  algorithms 

7.  Analysis  techniques  provided/supported:  List  and  explain 

®  MAUT  as  evidenced  by  SMART,  and  regression  based  policy  specifying 

8.  For  tools  using  Bayesian  Networks  (Ref.  Col  B  data)  are  there  built  in  methods  to 
trim  the  network  or  otherwise  limit  computational  complexity  with  “relevance 
reasoning”  or  any  other  method:  Yes  Q  No  Q 

a.  If  yes,  explain: 

Performance  and  Extensibility 

1 .  Language  used  to  develop: 

a.  GUI:  Active  Server  Pages,  VBScript 

b.  Computational  Engine:  Active  Server  Pages,  VBScript 

2.  Is  source  code  generally  available  to  a  government  end  user:  Yes  □  No  □  Yes 
(If  no,  is  there  any  circumstance  where  the  source  code  could  be  made  available?) 

3.  Is  the  API  available  to  the  end  user:  Yes  □  No  □  No 

4.  Availability  of  the  tool’s  performance  measurement:  Yes  □  No  □  No 

a.  If  yes,  please  indicate  the  benchmark/procedure  used  to  measure  performance: 
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5.  Maximum  Nodes  a  single  model  can  support:  Max  nodes  a  model  can  support  when 
running  on  the  recommended  system  No  hard  limits.  Only  limited  by  physical 
memory. 

6.  Error  recovery  support  (if  something  crashes,  what  happens?):  Software 
automatically  saves  inputs  as  the  user  works.  So,  if  a  system  failure  occurs,  the 
user  can  pickup  exactly  where  they  left  off. 

Interoperability 

1.  Ability  to  interface  with  databases:  Yes  O  No  O  Yes 

a.  If  Yes, 

i.  Using  Open  Data  Base  Connectivity  (ODBC):  Yes  Q  No  Q  Yes 
1 .  If  Yes,  list  the  data  bases  the  ODBC  driver  supports: 

MS  Access 

ii.  Databases  it  can  interface  directly  with(list  them):  None 

b.  Does  it  support  SQL  interface:  Yes  Q  No  Q  Yes 

c.  Can  models  created  in  the  tool  be  saved  to  a  database:  Yes  Q  No  Q  Yes 

2.  Format  used  to  the  save  the  model:  ??? 

3.  Export  the  model  to  other  applications:  Yes  Q  No  Q  No  (?) 

a.  If  yes,  list  the  applications  the  model  can  be  exported  to: 

4.  Ability  to  export  results/graphs  generated  from  the  “what-if  ’  scenarios  or  the 
analysis:  Yes  I  I  No  Q  No 

a.  If  Yes,  list  applications  that  it  can  be  exported  to:  Eg:  PPT,  Word,  JPEG,  PDF 
format,  PS  format,  etc 
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5.  Tools  uses  open  standards:  Yes  Q  No|  I  ??? 

a.  If  yes,  explain: 

6.  Provide  group  collaboration  on  model  building  and  analysis:  Yes  Q  No  Q  Yes 
(e.g.,  can  nodes  be  added  as  “sub-models”  created  by  another  user?)  No 
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Tool  Name:  Netica 

Website:  www.norsys.com 

Company’s  Name:  Norsys  Software  Corp. 

General  Features 


1 .  Cost:  Single  licenses:  $585  for  GUI,  $685  for  API.  Site  licenses  available  for 
5  times  as  much.  API  embedded  is  $20  to  $175  depending  on  volume. 

2.  Maturity  of  the  tool:  In  development  since  1992;  sold  since  1995 

3.  Targeted  Industries:  Industrial  diagnosis,  financial  risk  management, 
enviromental  planning,  decision  analysis,  user  modeling. 

4.  Major  clients:  Exxon,  Boeing,  AIG  Risk  Finance,  Lockheed  Martin, 

Electricite  de  France,  CIA,  MIT,  Stanford,  Motorola,  NASA,  SAIC,  Siemens, 
Rockwell,  US  Navy,  Northrop  Grummon,  Raytheon  and  many  others  (see 
www.norsys.com/clients.htm) 

5.  Type  of  Operating  system  supported:  Windows  95  to  XP,  MacOS,  and  API  for 

Linux,  Sun  Solaris  and  HP-UX. 

6.  Recommended  Hardware: 

a.  Processing  power:  Practical  with  200  MHz  but  large  models  require 
more 

b.  RAM:  Practical  with  128  MB  but  large  models  require  more 

c.  Other: 

7.  Architecture  support: 

a.  Client  -  Server  Model:  Yes  ^  No  Q 

b.  Desktop:  Yes  [>3  No  |  I 

Additional  Comments: 
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Input  Manipulation 


1 .  Type  of  Network  support: 

a.  Bayesian  Networks:  Yes  13  No  I  I 

b.  Decision  Networks/Influence  Diagrams:  Yes  3  No  EH 

c.  Both  Bayesian  Nets  and  Influence  Diagrams:  Yes  3  No  fH 

2.  Model  Building  capability: 

a.  Using  GUI:  Yes  |EI  No  □ 

b.  From  databases(i.e.  Structure  Learning):  Yes  I  1  No  13 

c.  Other:  Currently  adding  structure  learning 

3.  Type  of  graph  support: 

a.  Only  Undirected  graph:  Yes  HD  No  EH 

b.  Only  Directed  graph:  Yes  13  No  I~1 

c.  Chain  graphs  (i.e.  mixed  undirected  and  directed):  Yes  I  1  No  I  I 

4.  Conditional  Probability  Table: 

a.  Specify  Tables:  Yes  3  No  [3 

b.  Specify  Equations:  Yes  3  No  EH 

c.  From  databases  (learn  from  training  sets):  Yes  13  No  I~1 

d.  Other:  Sequential  updating,  and  missing  data  can  be  handled  by  the  EM 
learning  or  gradient  descent  algorithms 

5 .  Validity  of  the  model : 

a.  Consistent  Probability  check  of  variables  in  the  net:  Yes  13  No  f~l 

b.  Cyclic  Dependency  structure  check  among  variables:  Yes  3  No  EH 

6.  Inference  Algorithm  used:  Several,  mainly  junction  tree  (aka  known  as  join 
tree  or  clique  tree).  Logic  sampling. 

7.  Analysis  techniques  provided:  Sensitivity  to  findings  (mutual  information, 
variance  reduction,  etc),  Testing  the  net  for  accuracy  with  a  database  of 
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cases,  "Processing"  sets  of  cases  using  the  network,  "Summing-out"  sections  of 
the  network,  noisy-or,  noisy-and,  noisy-max  and  noisy-sum  nodes 

8.  For  tools  that  support  Bayesian  Networks  (Ref.  Question  1)  are  there  built  in 
methods  to  trim  the  network  or  otherwise  limit  computational  complexity  with 
“relevance  reasoning”  or  any  other  method:  Yes  [Xl  No  1  I 

a.  If  yes,  explain:  Sensitivity  analysis,  "summing  out"  parts  of  the  network 

Additional  Comments: 


Performance  and  Extensibility 

1 .  Language  used  to  develop: 

a.  GUI:  C++ 

b.  Computational  Engine:  C++ 

2.  Is  source  code  available  to  the  end  user:  Yes  No  [x] 

3.  Is  the  API  available  to  the  end  user:  Yes  [X]  No  I  I 

4.  Availability  of  the  tool’s  performance  measurement:  Yes  [X]  No  Q 

a.  If  yes,  please  indicate  the  procedure  in  measuring  the  performance: 

Test  for  prediction  or  diagnosis  accuracy  against  a  database  of  cases:  Error 
rate,  logarithmic  loss,  Brier  score,  confusion  matrix,  surprise  matrix,  ROC 
curves,  etc. 

5.  Maximum  Nodes  a  single  model  can  support:  No  limit;  Easily  handles  1000s  if 
the  graph  structure  is  simple. 

6.  Error  recovery  support:  Yes  I  I  No  I  I 

a.  If  yes,  explain: 
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Additional  Comments:  Extensive  effort  has  been  put  into  error  checking,  and 
recovering  well. 


Interoperability 

1 .  Ability  to  interface  with  databases:  Yes  CR1  No  [~~l 

a.  If  Yes, 

i.  Using  ODBC:  Yes  |EI  No  □ 

ii.  Databases  it  can  interface  directly  with  (list  them):  Windows 
ODBC  databases,  such  as  MS  Access 

b.  Does  it  support  SQL  interface:  Yes  fxl  No  I  I 

c.  Can  the  models  be  saved  to  a  database:  Yes  I  1  No  Ex] 

2.  Format  used  to  the  save  the  model:  DNE,  NETA,  can  also  read  several  other 
formats,  such  as  Ergo,  DX  Express,  Hugin.  XML  implemented  but  not  yet 
available. 

3.  Export  the  model  to  other  applications:  Yes  I  I  No  I  I 

a.  If  yes,  list  the  applications  the  model  can  be  exported  to:  Some  other 
programs  can  read  Netica  format,  but  we  don't  have  any  experience 
or  knowledge  of  how  well. 

4.  Ability  to  export  results/graphs  generated  from  the  “what-if  ’  scenarios  or  the 
analysis:  Yes  Q  No  ^ 

a.  If  Yes,  list  applications  that  it  can  be  exported  to:  Nice  graphics  of  the 
net  can  be  copied  and  pasted  into  Microsoft  office  products.  Tables  of 
conditional  probabilities  and  inference  results  can  be  pasted  into 
Excel  or  a  text  file  (or  read  from  them). 

5.  Tools  uses  open  standards:  Yes  I  I  No  I  I 

a.  If  yes,  explain: 
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6.  Provide  group  collaboration  on  model  building  and  analysis:  Yes  I  I  No  1^ 


,  Additional  Comments: 
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DARPA’s  Wargaming  The  Asymmetric  Environment  (WAE)  Program 

What  is  WAE’s  mission?  The  mission  of  the  WAE  program  is  to  develop  and 
demonstrate  models  and  tools  “tuned”  to  specific  adversaries,  thus  enabling  analysts  and 
decision-makers  to  better  anticipate,  predict,  and  act  against  those  who  threaten  U.S.  and 
Allied  interests.  Asymmetric  adversaries,  for  WAE’s  purpose,  span  the  continuum  from 
guerrilla  warfare  as  experienced  in  the  Balkans  to  the  present  terrorist  threat.  The 
resultant  technology  will  yield  an  automated  Continuous  Indication  and  Warning  System 
composed  of  multiple  group-specific  predictive  models  (FY  04). 

What  are  WAE’s  research  questions?  The  research  questions  for  WAE  are  numerous 
and  include  at  a  high  level;  do  pre-incident  indicators  exist  that  provide  relevant 
predictions  of  our  adversaries’  future  behavior?  If  predictive  pre-incident  indicators 
exist,  what  predictive  fidelity  do  they  possess?  Can  these  pre-incident  indicators  predict 
the  timeframe,  target,  direction  of  interest,  or  tactical  characteristics  of  future  attacks? 
What  is  the  shelf  life  of  these  pre-incident  indicators?  Can  these  pre-incident  indicators 
provide  any  insight  into  how  to  influence  these  adversaries?  As  quick  disclaimer,  WAE’s 
technology  does  not  possess  the  ability  to  predict  the  specific  day,  time,  target  address, 
and  method  of  an  attack.  However,  WAE  does  contend  that  prediction  of  an  adversary’s 
behavior  is  possible  at  a  detail  level  that  at  a  minimum  dramatically  increases  the 
specificity  of  the  indication  and  warning  space.  The  metrics  for  assessing  WAE’s  success 
are  straightforward;  the  predictive  technology  is  frequently  validated  against  both 
historical  information  and  real-time  information. 

How  does  WAE’s  approach  differ  from  the  current  analytical  approach?  WAE’s 
approach  is  vastly  different  from  the  current  analytical  approach  and  supporting 
technologies.  This  difference  is  best  articulated  in  an  analogy  of  attempting  to  predict  the 
behavior  of  an  electronic  circuit  of  unknown  design  contained  within  a  black  box.  The 
current  analytical  method  and  tools  will  attempt  to  re-create  the  design  of  the  circuit 
within  the  black  box;  understanding  the  major  components  and  their  respective 
interactions.  WAE’s  approach,  on  the  other  hand,  concentrates  on  the  environment 
external  to  the  black  box,  specifically  the  input  and  output  behaviors.  Thus  WAE’s 
approach  is  to  derive  the  behavioral  transform  function  by  analyzing  the  relationship 
between  the  input  and  output  signals  across  a  range  of  inputs.  The  result  is  a  transfer 
function  that  predicts  the  circuit  behavior  without  replicating  the  circuit  design. 

Why  is  this  approach  important?  This  approach  is  important  for  three  reasons.  First, 
WAE’s  approach  provides  reliable  predictions  of  near-term  future  behavior  (continuous 
indication  &  warning)  while  the  existing  detection  approach  and  technology,  at  best, 
provides  a  description  of  recent  past  behavior.  Second,  WAE’s  approach  uses  a  different 
and  more  readily  available  information  base  than  current  detection  approaches  and  can 
thus  derive  predictive  models  when  detection  approaches  falter.  An  example  of  this  is 
the  Greek  terrorist  organization  referenced  in  the  results  table  above.  This  group 
operated  for  27  years  without  the  US  intelligence  community  knowing  any  group 
members.  Without  knowledge  of  the  group  members,  the  US  intelligence  information 
consisted  largely  of  forensic  information.  WAE’s  approach  derived  pre-incident 
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indicators  that  were  stable  for  the  early,  middle  and  final  years  of  this  group.  Third,  the 
pre-incident  indicators  that  drive  WAE’s  predictive  models  are  in  form  that  is  consistent 
with  and  usable  by  the  information  operations  community,  with  the  added  benefit  of  also 
provided  observable  metrics. 

What  are  WAE’s  predictive  results?  WAE  has  derived  models  of  a  number  of 
asymmetric  groups  in  conjunction  with  our  operational  partners.  WAE’s  validates  its 
predictive  models  through  both  historical  and  real-time  tests  against  the  operation  target. 
An  example  of  WAE’s  initial  results  is  represented  by  recent  tests  of  a  predictive  model 
of  a  specific  European  Terrorist  organization.  The  model  was  derived  from  a  training  set 
of  attacks  and  non-attacks  that  spanned  the  life  of  this  organization.  The  results  are 
presented  here  as  the  percentage  of  correct  predictions  along  two  dimensions.  First  is  the 
fidelity  of  the  attack  characteristics:  attack/no  attack,  target,  direction  of  interest,  and 
tactical  characteristics.  The  second  is  the  predictive  accuracy,  true  positive  and  true 
negative.  True  positive,  for  example,  represents  the  percentage  of  correct  predictions  that 
the  next  attack  would  reflect  the  nature  of  each  attack  characteristic.  Conversely,  true 
negative  represents  the  percentage  of  correct  predictions  that  the  next  attack  would  not 
reflect  the  nature  of  each  attack  characteristic.  For  example,  if  the  model  predicts  the 
next  attack  is  against  a  US  asset,  it  is  also  predicting  the  next  attack  will  not  be  against 
the  Host,  Adjacent,  or  other  International  countries. 

As  you  can  see  the  results  are  high  across  the  board  and  in  fact,  from  a  statistical 
perspective,  the  predictive  accuracy  for  each  attack  characteristic  is  significant  well 
beyond  the  traditional  p<.01  criteria.  What  is  clear,  at  least  for  this  group,  is  the 
existence  of  discemable,  predictive  patterns  to  their  behavior  at  a  level  specific  enough  to 
support  the  indication  and  warning  process.  The  signatures  clearly  distinguish  between 
the  environmental  conditions  preceding  an  attack  versus  no  attack,  a  civilian  versus  a 
military  target,  a  privately  owned  versus  a  publicly  owned  target,  and  an  attack  on  a  U.S. 
versus  NATO  target.  Furthermore,  the  results  indicate  that  the  fixed  set  of  signatures  is 
predictive  over  the  life  of  this  group  as  well  as  over  the  evolution  of  this  group’s  tactical 
capability. 
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Unclassified  -  FOUO 


►  European  Terrorist  Group  I 

•  Historical  test  on  82  attacks 

•  Modeled  group  behavior.  Did  not  use  data 
on  specific  group  members,  money  or 
logistics 

►  Testing 

«  Model  derived  on  a  training  set  of  attack  & 
non-attack  data  spanning  the  life  of  the 
organization 

■  Model  tested  on  a  separate  but  equivalent 
data  set 

►  Results 

■  Predictions  of  each  response  are  statistically 
significant 

•  Predictions  are  stable  over  life  of  group 

•  Predictions  are  stable  over  the  group's 
tactical  evolution 


Next  Attack 


True  + 

True  - 

Attack 

99% 

93% 

No  Attack 

93% 

99% 

Next  Target 

Civilian 

100% 

100% 

Military 

100% 

100% 

Private 

89% 

80% 

Public 

80% 

89% 

Next  Direction  of  Interest 

U.S. 

86% 

93% 

Host 

93% 

100% 

Adj  acent 

75% 

99% 

International  67% 

95% 

Next  tactic 

Direct  Fire 

78% 

88% 

Indirect  Fire 

46% 

82% 

Bomb  mg 

68% 

73% 

Unclassified  -  FOUO 


Who  are  WAE’s  operational  partners?  Since  its  inception  (2000),  WAE  has  worked 
cooperatively  with  DoD  and  the  intelligence  community  to  develop  and  demonstrate 
predictive  models  and  tools  “tuned”  to  specific  adversaries,  thus  enabling  analysts  and 
decision-makers  to  better  anticipate,  predict,  and  act  against  those  who  threaten  U.S.  and 
Allied  interests.  Our  partners  include  the  Joint  Information  Task  Force  for  Counter 
Terrorism  (JITF-CT),  the  Joint  Staff  J3  -  Deputy  Director  for  Information  Operations 
(J39),  and  the  Joint  Information  Operations  Center.  These  partnerships  range  from 
supporting  an  on-site  WAE  team  that  works  closely  with  analysts  to  develop  and  test 
predictive  tools  to  providing  independent  review  of  predictive  results  and  technologies. 

As  a  result,  WAE  and  the  operational  community  have  developed  technologies  that 
derive  group  specific  pre-incident  indicators  of  impending  attacks,  converted  these  pre¬ 
incident  indicators  into  predictive  models,  and  then  validated  the  predictive  model  against 
both  historical  and  real-time  information.  As  a  result  of  these  successful  tests,  operational 
partners  such  as  the  JITF-CT  are  working  with  WAE  to  further  test  and  transition  the 
predictive  models  and  tools  into  their  analytical  process. 
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Bedford,  Massachusetts  09071 
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Dear  Mr.  Nakamoto, 


Please  find  enclosed  the  final  report  of  the  Market  Research  Survey  for  Time  Line  Analysis 
Tools  in  response  to  the  research  proposal  accepted  by  your  office.  We  have  prepared  the  final 
report  in  softcopy  in  two  formats  for  inclusion  in  your  Knowledge  Portfolio  and  the  MITRE 
IT  Advisor  as  you  requested. 


Thank  you  for  the  opportunity  to  conduct  this 
behalf  of  The  MITRE  Corporation.  I  am  looking 
working  with  your  office  on  future  efforts  of 
portfolio . 


research  on 
forward  to 
the  knowledge 


Respectfully, 


John  Wigle,  G066 
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Executive  Sum  man 


Since  September  1  L  2001  the  needs  of  the  United  States  Intelligence  Community  (1C) 
have  increased  lor  time  line  analysis  to  assist  with  counter-terrorism  analysis  and  the 
Asymmetric  Order  of  Battle  (ASOB)  problem.  MITRE  does  not  have  in-depth  experience 
with  time  line  analysis  tools  as  evidenced  by  their  lack  in  the  MITRE  Analysis  Tool  Shed 
and  other  internal  analytic  tool  collections.  Currently,  the  IC  is  in  the  process  of 
identifying  new  signatures  and  methods  of  detecting  and  predicting  terrorism.  Time  line 
analysis  is  and  will  be  an  important  tool  in  combating  terrorism  and  the  Asymmetric  Order 
of  Battle  (ASOB)  problem. 

To  better  understand  the  capabilities  of  time  line  analysis  tools,  and  how  they  support 
the  counter-terrorism  mission:  the  MITRE  Information  Technology  Center  (G060) 
initiated  an  internal  time  line  analysis  tool  study.  This  MITRE  study  was  conducted  during 
the  summer  of  2002.  The  MITRE  study  team’s  first  step  was  to  meet  with  several  MITRE 
and  U.S.  Government  intelligence  analysts  and  collect  a  set  of  functional  requirements  for 
a  time  line  analysis  tool.  Using  these  requirements  the  study  team  performed  a  market 
survey  of  available  products  that  may  meet  some  or  all  of  the  requirements.  The  study 
performed  a  paper  comparison  of  the  tools  followed  by  vendor  demonstrations  and 
requests  for  demonstration  time  line  analysis  to  be  added  to  the  MITRE  Analyst  Tool 
Shed.  The  results  of  the  study  are  summarized  by  a  scored  comparison  matrix  of  the  best 
lime  line  analysis  tools  currently  available. 

Based  upon  the  requirements,  the  METRE  study  team’s  research,  and  the  information 
provided  by  the  vendors,  our  recommendation  is  i2’s  Analyst  Notebook  version  6.  It  was 
the  most  intuitive,  was  verified  to  have  had  most  of  the  high-level  requirements,  and  had 
helpful  documentation.  The  i2  staff  was  also  very  helpful  and  quick  to  respond  to 
questions  and  support.  Unfortunately  it  was  not  due  out  until  the  end  of  the  2002  calendar 
year. 

The  other  highly  ranked  tool  to  examine  closely  would  be  VisuaLinks  by  Visual 
Analytics.  And  finally,  watch  for  a  Windows  release  of  CaseTrak  III  by  Badge  1022 
Software.  The  tool  remained  promising,  and  showed  well  fora  shareware  product. 
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/.  OVERVIEW 
LI  Background 

Since  September  11, 2(X)1,  the  analytical  needs  of  the  United  States  Intelligence 
Community  (1C)  have  increased  to  identify  new  signatures1  and  methods  of  detecting  and 
predicting  global  terrorism.  There  are  many  areas  of  automated  analysis,  including  entity 
extraction,  text  mining,  categorization,  summarization,  and  search  and  retrieval2.  However 
time  line  analysis  has  been  largely  overlooked  At  the  time  of  printing,  time  line  analysis 
tools  remained  a  weak  point  among  the  MITRE  capability  as  evidenced  by  the  MITRE 
Analyst  Tool  Shed.3 

After  9/1 1  several  customers  engaged  in  operational  analysis  requested  MITRE  to 
integrate  analytical  tools  into  their  environments  and  requested  easy  to  use  tools4.  Among 
the  areas  requested  in  automated  analysis  was  time  line  analysis.  Additionally,  several 
MITRE  analysts  and  an  engineer5  were  interested  in  various  automated  tools,  including 
time  line  analysis.  It  became  apparent  very  quickly  that  time  line  analysis  was  an 
important  tool  in  combating  the  Asymmetric  Order  of  Battle  ( ASOB)  problem. 
Additionally,  we  discovered  that  every  analyst  had  a  different  need  and  was  grappling 
issues  with  different  levels  of  complexity  ranging  from  simple  to  robust.  Having  tools  at 
varying  degrees  of  complexity  would  be  critical  to  providing  the  most  effective  means  of 
automated  analysis6. 

The  Information  Technology  Center  core  technology  program  funded  this  study.  The 
performance  of  this  study  was  pait  of  the  ITC’s  Knowledge  Portfolio  Program  (MITRE 
Information  Intranet  link:  http://g060.mitre.org/network_site/index.html). 

1.2  Objectives 

Through  market  research  and  evaluation  of  time  line  analysis  tools,  explore  the  current 
state  of  affairs  in  the  time  line  analysis  market  to  better  enable  MITRE  to  respond  to  the 
analytical  community,  and  to  provide  the  analytical  community  a  better  range  of  tools 
from  which  to  select  an  automated  solution.  Specifically  to: 

•  Determine  the  high-level  automated  tool  requirements  of  analysts 

•  Find  new  time  line  analysis  tools  not  listed  in  the  MITRE  Analysis  Tool  Shed 

•  Evaluate  tools  against  the  requirements  identified  by  the  analysts. 


*  Jerry  Cogle.  “TimeJine  Analysis  Tools.*’  E-mail  lo  Julie  Gravallesc  and  Glen  Nakumotn.  22  March  2002. 

3  Glen  Nakamoio’s  Transfer  Folder  -  Knowledge  Portfolio  <hitp//iranster.miirc.org/STF40- 

49/42/1 3742/Transter/Knowledge%20Portlolio/Products/TopicMapColla!>orationWcbP!csomation/diK/PKM_E\port- 
1 95.htm>  August  3 1 . 2002 

'  Analysis  Tool  Shed  home  page,  <http://idia.miire.org/toolshcd/index.htm>  August  31. 2002. 

A  Mark  Maybury  “Re:  Timeline  Analysis  Tools.”  E-mail  to  Jerry  Cogle.  Rod  Holland,  and  John  Ciriflith.  01  April  2002. 

5  Tom  Carroll  (G073),  Barry  Costa  (G05I )  and  Tom  McEniee  (G075) 

6  Time  Line  Analysis  Market  Research  and  Investigation  Proposal.  G06f«  dated  March  22.  2002. 
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1.3  Approach 

The  study  team  met  with  and  interviewed  two  analysts  from  The  MITRE  Corporation7 
and  tram  the  U.S.  Government s.  The  analysts  interviewed  represented  a  variety  of  analysis 
disciplines  with  over  80  years  ol  combined  experience.  From  these  interviews  the  study 
team  developed  high-level  requirements  for  time  line  analysis  tools. 

The  interviews  conducted  were  primarily  focused  on  identifying  the  manual  processes 
used  that  did  not  incorporate  computers  or  other  automated  processes.  This  methodology 
was  employed  so  that  analysts,  who  are  not  currently  using  automation,  had  the 
opportunity  to  express  their  needs  and  desires  for  an  ideal  computer  application.  By 
deriving  requirements  from  the  analyst  s  perspective,  the  study  team  hopes  to  achieve 
acceptance  from  all  analysts  in  the  use  of  the  tools  evaluated  meeting  the  stated 
requirements. 

The  evaluation  ot  tools  was  based  on  the  high  level  requirements  identified  in  the  next 
session.  We  developed  an  Excel  spreadsheet  detailing  the  high  level  requirements,  what 
we  looked  for  within  that  high  level  requirement,  and  indicated  for  each  tool  whether  it 
met  the  requirements  or  not.  The  vendors  were  contacted  and  given  a  copy  of  the  high 
level  requirements  and  asked  to  comment  on  them,  and  provide  a  demo  or  evaluation  copy 
oi  their  tool.  The  study  team  made  a  concerted  effort  to  confirm  the  vendor’s  remarks  on 
their  tool’s  ability  to  satisfy  the  stated  requirements  through  direct  demonstration  of  the 
tool’s  capability. 

In  the  event  a  vendor  did  not  respond  to  our  inquiries,  the  study  team  did  their  best  to 
evaluate  the  demo  without  the  vendor’s  assistance.  And  if  the  study  team  was  unable  to 
verify  the  capability  through  a  demo  or  evaluation  copy,  then  the  vendor’s  remarks  were 
relied  upon  as  truthful  but  were  notated  that  they  were  unconfirmed.  If  the  vendor  neither 
responded  nor  provided  a  direct  demonstration  of  the  capability  to  the  team,  then  the  tool 
is  considered  not  evaluated  for  this  report. 

1.4  High  Level  Requirements 

The  very  basic  requirement  for  this  research  effort  was  a  software  utility  or  tool  that 
was  able  to  produce  a  time  line  with  associated  information  to  allow  the  user  to  conduct 
meaningful  analysis,  assessments  and  connections.  The  study  team  used  the  expounded 
requirements  as  a  means  of  creating  the  Tools  Comparison  Chart.  The  high  level 
requirements  for  a  time  line  analysis  tool  provided  by  the  intelligence  analysts  were; 


Table  1  -  High  Level  Requirements 


1 

Automation  of  input  from  various  sources  and  in  various  formats. 

2 

Save  the  customized  method  of  data  import  for  reoccurrmg  formats. 

3 

Suve  bibliographical  data  references,  kind  of  events,  dates  and  times  for  events  and  ! 

k  Tom  (G073)  and  McEntee.  1‘om  <G075).  Personal  interviews.  5  April  2002  and  26  April  2002,  respectively. 

Unidentified  intelligence  analyst  of  the  US  Government  Personal  interview.  30  April  2002.  And  an  unidentified.  Spcciul 
Agent  with  the  U.S  Department  ot*  Agriculture.  Criminal  Investigation  Unit.  Personal  interview-.  14  May  2002. 
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! - 1 

l- . . . 

entities. 

1 

4 

Comment  on  an  event  or  entity,  analogous  to  a  post-it'  note. 

5 

Query  events  and  entities  by  time.  dale,  references,  and  kind  of  event 

6 

Place  events  on  multiple  threads. 

7 

Gantt  chart  known  problems  to  reverse  engineer  events. 

8 

Move  events  around  time  line. 

9 

Expand  and/or  compress  time  line. 

10 

Categorize  events  with  icons. 

7 

11 

Export  to  various  formats,  including  power  point.  MS  word.  Visio,  web,  PDF 

12 

Output  to  large  plotter  devices 

13 

Has  an  intuitive  and  user-friendly  interface 

1.5  Meanings  Used  for  High-Level  Requirements 

Automation  of  input  from  various  sources  was  divided  into  three  areas,  manual,  semi¬ 
automatic  and  dynamic.  Manual  meant  the  user  selected  the  appropriate  menu  items  to 
initiate  a  manual  process  of  importing  data  from  a  source  into  the  tool.  Semi-automatic 
meant  the  tool  had  a  wizard-like  interface  that  assisted  the  user  with  importing  data. 
Dynamic  meant  there  existed  a  facility  to  connect  to  other  data  sources  similar  to  ODBC 
or  XML,  etc. 

Saving  customized  data  import  formats  was  referred  to  as  a  filter.  A  filter  was  defined 
as  a  tool’s  ability  to  remember  the  data  format.  Similar  to  an  XML  DTD,  or  an  ODBC 
driver,  the  tool  could  understand  the  data  being  imported  once  an  appropriate  filter  was 
selected.  The  Department  of  Defense  (DoD)  Regulations9  requires  all  DoD  intelligence 
agencies  to  have  the  ability  to  understand  SGML,  HTML,  and  XML  for  data  exchange  and 
interoperability.  If  a  tool  was  ODBC  or  XML  enabled,  it  was  presumed  to  have  met  the 
first  two  high-level  requirements. 

Saving  bibliographical  data  references,  kind  of  events,  dates  and  times  for  events  and 
entities  was  broken  down  into  bibliographical,  time,  date,  and  references.  The 
bibliographical  data  meant  the  analyst  would  be  able  to  see  the  evidence  or  citation  to  raw 
intelligence  establishing  the  event  or  entity.  The  kind  of  event  meant  the  user  would  be 
able  to  make  a  notation  of  the  type  of  event,  like  a  drug  transaction,  wire  transfer,  phone 
cal],  etc.  and  is  named  references  in  the  second  column  of  the  Tool  Comparison  Chart. 
Time  and  date  are  self  explanatory,  except  that  it  w  as  important  for  analysis  to  be  able  to 
leave  dates  blank  if  they  did  know  when  an  event  occurred. 

The  Post-it^  note  requirement  was  liberally  construed  to  include  tools  that  had  the 
ability  to  place  a  text  box  next  to  an  event,  or  place  a  comment  or  note  in  the  event  itself. 

Querying  events  and  entities  by  time,  date,  references  and  kind  of  event  meant  the  tool 
had  the  ability  to  sort,  select  or  only  list  events  or  only  list  entities  matching  the  search 
criteria.  The  meaning  of  references  for  this  high-level  requirement  was  semantically 

v  DoD  Joint  Technical  Architecture  version  4.  dated  17  July  2002.  $  2.2.2. 1.4.1.  “Document  Interchange"  *hup://w*w- 
jta.iui.disa.mil/jta/JTA40_07l702.pdr> 
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different  from  high-level  requirement  number  3.  In  this  requirement,  references  meant 
bibliographical  references  The  search  criteria  had  to  allow  the  user  to  search  by  the  four 
areas  de lined,  whether  the  tool  used  a  sort,  a  select  or  a  listing.  The  four  areas  defined 
were  time,  date,  references,  and  type  ot  event. 

Placing  events  on  multiple  threads  meant  the  tool  had  the  ability  to  display  several 
time  lines  or  parallel  activities  on  the  same  time  domain  or  display. 

The  ability  to  Gantt  chart  known  problems  to  reverse  engineer  events  meant  the  tool 
had  the  ability  to  place  events  into  an  alternate  view  showing  tasks  or  threads,  schedules  or 
timelines  and  then  dependencies  like  a  Gantt  chart. 

Moving  events  around  the  time  line  meant  a  user  could  directly  manipulate  an  event  to 
either  expand/compress  the  time  line  on  each  side  the  event,  or  that  the  event  could  be 
moved  vertically  along  a  horizontal  time  line  to  facilitate  spatial  or  other  notation  for  the 
analyst. 

To  expand  and/or  compress  time  line  meant  the  user  could  universally  define  several 
different  scales  along  the  same  time  line. 

Categorize  events  with  icons  meant  that  the  user  had  the  ability  to  select  and  assign 
meaningful  icons  for  events  and  entities.  Icons  were  liberally  construed  to  include 
photographs  in  JPEG  and  GIT  format  oi  clip  art  in  addition  to  traditional  icons. 

Exporting  to  various  formats,  including  Microsoft  Power  Point,  Microsoft  Word. 

Visio.  web,  and  PDF  meant  precisely  that  except  web  meant  any  of  the  formats  commonly 
used  on  the  Internet  to  exchange  documents  and  graphics.  The  formats  for  the  web 
specifically  included.  HTML.  JPEG,  GIF,  and  text. 

To  ensure  coverage  by  DoD  and  non-DoD  agencies  and  components,  tools  were 
further  examined  to  see  if  they  could  read  from  major  data  sources  like  Oracle,  Microsoft 
Excel,  Sybase.  Microsoft  Access.  Informix,  Ingress,  and  Microsoft  SQL  Server10. 

Output  to  large  plotter  devices  meant  that  the  tool  was  capable  of  working  with  larger 
printer  devices  like  a  D-size  plotter  with  the  appropriate  windows  drivers  installed  on  the 
system,  or  at  least  was  able  to  tilt*  the  chart  on  regular  printer  size  paper  so  it  could  be 
assembled  into  a  large  chart. 

Had  an  intuitive  and  user-friendly  interface  was  a  subjective  requirement,  but  the  study 
team  did  its  best  to  be  objective.  User  friendly  and  intuitive  meant  that  the  tool  did  not 
require  the  user  to  open  the  help  files  and  manuals  too  often.  Specifically,  if  a  tool  was 
determined  not  to  be  intuitive  then  it  meant  the  tool  did  not  follow  the  Microsoft  Windows 
graphical  user  interface  standards  or  had  significant  problems  preventing  the  user  from 
completing  tasks. 


11  Joim  Technical  Architecture  List  oi  Mandated  and  bmcnzing  Standard*,  dated  17  July  2002  <http7/www  jta. itsi.disa.mil/> 
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1.6  Scoring 

Vendors  provided  information  on  the  high  level  requirements,  and  study  team 
members  evaluated  the  tools  individually  and  notes  were  compared  and  discussed.  There 
were  three  scores  associated  for  each  tool.  They  were  the  requirements  score,  the 
confidence  score  and  the  overall  score.  Each  score  is  normalized  between  0  and  10  with 
zero  meaning  very  poor  and  10  meaning  outstanding.  The  requirement  score  w  as 
computed  by  adding  up  the  total  number  of  defined  requirements  satisfied,  then  dividing  it 
by  the  total  number  of  defined  requirements.  The  total  number  of  defined  requirements 
was  twenty-six  (26).  Then  multiplying  it  by  10  for  a  normalized  score. 

The  confidence  score  ranked  the  study  team's  level  of  confidence  in  the  information 
they  had  about  the  tool.  If  the  study  team  was  unable  to  verify  if  a  requirement  was 
satisfied,  then  a  “not  reported"  is  shown  for  the  defined  requirement  on  the  Tool 
Comparison  Chart.  However,  if  the  vendor  asserted  they  met  the  requirement,  but  the 
study  team  was  finable  to  verify  the  fact,  then  a  ‘  ves  "  or  “nof”  is  listed  with  a  caveat  that 
it  was  considered  satisfied  only  based  upon  the  vendor’s  statements  without  verification 
by  the  study  team.  The  confidence  score  is  computed  by  totaling  the  number  of  defined 
requirements  not  verified  by  the  study  team,  then  subtracting  that  number  from  the  total 
number  of  defined  requirements  and  multiplying  the  result  by  10  to  normalize  the  score. 

Two  products,  VisuaLinks  and  TimeMap.  had  the  vendor  statements  listed.  The  study 
team  inquired  late  with  the  makers  of  VisuaLinks  about  the  high-level  requirements  and  a 
request  for  an  evaluation  copy.  There  was  sufficient  lime  to  conduct  a  cursory  review'  of 
their  tool,  but  the  vendor  prevented  the  study  team  from  beginning  an  evaluation.  The 
vendor  had  a  very  iterative  customer  service  process  in  obtaining  the  answers  to  the 
requirements,  then  for  permission  to  download  an  evaluation  copy,  then  to  obtain  a 
password,  and  finally  a  server  demo  license  key11.  T  imeMap  was  revisited  following 
feedback  on  the  final  report  and  vendor  statements  were  added  to  the  matrix.12 

The  overall  score  is  simply  computed  by  adding  the  requirements  score  with  the 
confidence  score,  and  again  was  normalized  hv  dividing  by  2.  This  system  of  scoring  is  far 
from  perfect  but  provides  a  reasonable  w'ay  to  compare  the  tools  w  ith  one  another  and  this 
type  of  method  closely  matched  the  study  team  s  expert  judgment12. 


11  Lynee  Weston.  Visual  Analytics  “RE:  New  Client  Profile  lor  jvvigleti?  mitiv.org"  E-mail  to  John  Wigle.  20  August  2002. 
13  Bob  Wiss.  Cascsolt  Software.  Telephone  Interview.  8  October  2002 

13  Dr  Lehner  (C306A)  Meeting  on  expert  judgment  tor  National  Intelligence  Priorities  Framework  20  July  2002. 
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II.  EVALUATIONS 

2.1  Tools  Investigated 

The  tooK  were  found  by  using  the  MITRE  Analyst  Tool  Shed  and  search  engines  over 
the  Internet.  The  tools  evaluated  were: 

•  Time  Map  by  Case  Soft 

•  WebTAS  by  Air  Force  Research  Labs,  Rome,  NY 

•  Watson  by  Xanalys 

•  Orion  VIA  by  Orion  Scientific  Systems 

•  Analyst  Notebook  by  i2  Inc 

•  Pen-Link  Analysis  Software  by  Pen  Link 

•  CaseTrak  III  by  1022  Badge  Software 

•  Vi  sna  Links  by  Visual  Analytics 

2.2  Discussion  on  Evaluated  Tools 

A  total  of  eight  tools  were  evaluated  for  the  market  survey  and  research  project.  Tools 
appear  in  the  same  order  as  the  Tool  Comparison  Chart.  The  discussion  covers  the  tool’s 
rank,  strengths,  weaknesses,  customer  support  and  a  recommendation.  Unique  attributes  or 
special  considerations  of  the  tool  were  included  for  completeness. 

2.2.1  l  ime  Map 

TimeMap  developed  by  Case  Soft  sells  for  an  affordable  $199  per  desktop.  It  was 
designed  to  work  with  companion  products  CaseMap  and  TextMap  used  by  law  firms, 
private  practice  attorneys,  and  prosecutors  to  prepare  their  cases  for  litigation  in  court. 
CaseSoft  had  also  sold  site  licenses  to  the  Department  of  Justice  and  Department  of 
Defense  Criminal  Investigation  communities  .  TimeMap  was  a  simple  way  to  make  a 
timeline  out  of  known  events1  \  It  scored  an  overall  7.7  in  a  3rd  place  tie  with  the  Watson 
tool.  It  was  above  average  on  meeting  requirements  with  a  score  of  6.5.  It  was  easy  to 
learn,  can  be  sell  taught  or  taught  by  demo  over  the  telephone1*.  It  was  one  of  the  lew' 
tools  that  had  a  Gantt  chart  and  plotter  capability,  along  with  better  than  average  ability  to 
export  to  other  formats.  TimeMap  had  a  semiautomatic  method  for  importing  data  from 
CaseMap,  which  in-turn  could  semi-uutomaticaUy  populate  from  TextMap17  It  Jacked  an 
ability  to  query  against  the  time  line.  It  could  not  import  data  from  other  sources 
automatically  for  populating  a  timeline  but  simply  makes  one  out  of  events  manually 
inputted15".  Expect  good  customer  support.  They  were  quick  to  respond,  had 
demonstration  software  on  their  website,  and  were  helpful  in  answering  any  questions. 
Recommended  for  analytical  shops  looking  for  modest  capability  at  a  modest  price. 


14  Thomas  McEntee  “Re:  Time  Line  Analysis  Report  .  ‘  E-mail  to  John  Wigle.  29  September  2002 
Jeffrey  Lisson.  Attorney.  “Evidence  Analysis  Timelines  Are  Not  Just  for  Trial  Anymore'  Legal  Tech  April  2002  Vol.  20, 
No  I.Pg.  3. 

16  Jennifer  Wchslct  “added  inlo  about  casesolt.  '*  E  mail  to  John  Wigle.  29  August  2002. 

!  Thomas  McEmec.  “Re.  Time  Line  Analysis  Report..."  E-mail  to  John  Wigle.  29  September  2002 
,K  Thomas  McEntce.  “CaseSot't  Legal  Software  Application'”.  E-mail  to  John  Wigle.  17  May  2002. 
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2.2.2  WebTAS 

WebTAS  developed  by  Air  Force  Research  Labs  (AFRL)  stas  customized  rapid 
development  software.  It  was  designed  for  govern  mentor  large  enterprise  customers.  It 
scored  an  overall  7.9  as  the  2nd  place  tool,  li  was  better  than  average  on  meeting 
requirements  at  a  score  of  6.2.  It  was  one  of  the  few  tools  that  had  a  Gantt  chart  and  could 
dynamically  access  other  data  sources.  WebTAS  was  looked  at  heavily  as  an  import 
system  that  automatically  put  events  on  time  lines.  There  were  some  powerful  capabilities, 
such  as  geographic  mapping,  that  were  not  displayed  by  any  commercial  products.  The 
events  were  highly  searchable,  but  the  learning  curve  was  high  to  retrieve  information.  A 
two-day  specialized  training  session  was  required  at  $1,535  just  to  begin  working  with  this 
tool19.  The  manual  was  overly  specific  in  some  areas  and  vague  in  others20.  The  tool  itself 
was  not  intuitive  to  use21.  Its  weaknesses  were  definitely  an  unfriendly  interface,  a  lack  of 
ability  to  export  to  other  formats,  and  less  annoying  but  simple  features  like  placing 
comments  or  notes  on  events22,  and  moving  the  events  around  on  the  time  line.  The  study 
team  and  others  at  MITRE  were  unable  to  get  WebTAS  working22,  and  relied  on  Janet 
Hitzeman’s  work  with  WebTAS  to  substantiate  some  of  its  capabilities24.  Customer 
support  from  AFRL  is  weak.  AFRL  were  no-shows  at  or  cancelled  several  meetings  with 
potential  customers  before25.  It  would  be  essential  with  this  product  to  pay  for  and  ensure 
software  development,  maintenance  and  user  training  were  provided  since  this  software 
would  be  built  uniquely  for  each  customer  base.  Recommend  only  for  large  shops  with 
political  clout  and  a  large  budget  to  support  software  development,  maintenance  and  user 
training  and  to  put  pressure  on  AFRL  if  they  fell  behind  in  their  support. 

2.23  Watson 

Watson  developed  by  Xanalys  was  a  commercial  solution  with  an  undisclosed  price 
tag.  It  was  designed  for  law  enforcement,  intelligence  and  government  customers.  It 
scored  an  overall  7.7  placing  it  in  a  3rd  place  with  TimeMap.  It  was  above  average  on 
meeting  requirements  with  a  score  of  6.5.  It  was  strong  on  user  features  and  export 
formats.  News  reports  claim  it  was  the  world's  best  analytical  tool,  and  it  was  voted  the 
best  intelligence  analysis  tool  by  an  international  law  enforcement  association  in  199926. 
However,  it  did  not  support  dynamic  access  to  other  data  sources  or  filters  to  simplify 
manual  data  importing.  Expect  less  than  average  customer  support  for  this  tool.  There  was 
no  response  to  inquiries  for  pricing  information  and  some  requirements.  Recommended 
for  small  to  medium  shops  that  can  afford  to  pay  for  premium  customer  support  if  offered. 


19  Jay  Jesse.  “Re:  WebTAS.**  E-mail  to  John  Wiglc.  09  February  2002. 

20  Janet  Hitzeman  “Re:  webTAS.”  E-mail  to  Patrick  Jones.  1 1  April  2002.  and  see  next  foot  note. 

21  Janet  Hitzeman.  “FW:  WebTAS*’  E-mail  to  Michael  Maskaleris.  03  May  2002. 

22  Janet  Hitzeman.  “FW:  WebTAS”  E-mail  to  Michael  Maskaleris.  03  May  2<X)2. 

23  Barry  Costa.  “Re:  WebTAS  ”  E-mail  to  Michael  Maskaleris  und  John  Wigle.  06  May  2002. 

24  John  Griffith.  “Re:  Timeline  Analysis  Tools.”  E-mail  to  Mark  Maybury.  Jerry  Cogle.  Rod  Holland.  Janet  Hitzeman, 
Michael  Meridcth.  Julie  Gravallese,  Aaron  Lesser.  John  Wiglc  and  Glen  Nakamolo.  01  April  2002. 

25  Herbert  Mucks,  WebTAS  PM  ”RE:  webtas.”  E-mail  to  John  Wiglc.  14  March  2002. 

26  PRLine  -  England.  “Global  Graphics  Announces  a  Euro  5.1  Million  Canadian  Government  Software  Contract  for  Xanalys 
Inc.”  2  November  2000. 
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2.2.4  Orion  VIA 

Orion  VIA  developed  by  Orion  Scientific  Systems  offered  this  commercial  solution 
originally  developed  in  cooperation  with  the  Defense  Advance  Research  Projects  Agency. 
It  was  designed  as  an  automated  intelligence  analysis  system,  and  was  professed  to  be  a 
fixture  in  the  1C.  It  scored  an  overall  7.1  placing  it  in  5,h  place.  It  was  average  in  meeting 
requirements  with  a  score  of  5.0.  The  tool  offered  a  Gantt  chart  and  large  plotter 
capability,  and  dynamic  import  from  other  data  sources.  Its  shortcomings  were  weak  query 
capability  against  the  time  line,  poor  data  export  formal,  and  no  ability  to  save  data  filters 
to  facilitate  common  data  imports  to  the  tool.  Expect  the  average  customer  support  from  a 
commercial  software  vendor.  Recommended  for  shops  to  review,  but  only  use  if  it  solves 
unique  needs  other  tools  do  not  address. 

2.2.5  Analyst  Notebook  6 

Analyst  Notebook  6  by  i2  Incorporated  was  due  out  at  the  end  of  2002,  and  was  a 
commercial  solution  with  $3388  price  lag  per  license  that  includes  1  year  of  technical 
support.  It  was  developed  for  the  British  1C,  and  had  customers  all  over  the  world, 
including  the  United  States2  .  We  got  a  sneak  peek  look  at  the  pre-release28  It  scored  an 
overall  9.0  taking  first  place  by  a  significant  margin.  It  was  outstanding  in  meeting 
requirements  with  a  score  of  8. 1 .  one  of  the  highest  on  the  evaluation.  The  tool  offered  a 
rich  user  feature  set.  rich  data  import  capability,  and  an  easy  to  use  interface.  It  could 
import  data  or  connect  directly  to  third  party  databases29  like  Microsoft  Access  and  Sybase 
through  its  1-Bridge  product.  Events  were  searchable,  and  useable  for  both  time  line 
analysis  and  link  analysis.  Icons  were  used  on  the  time  line  and  could  be  changed  or 
customized.  The  manual  contained  help  documentation  that  was  very  good  for  learning 
how  to  use  the  tool.  Additionally.  Analyst  Notebook  6  had  a  robust  Application 
Programming  Interface  (API)  to  allow  customization  of  the  tool  if  needed.  Its  weaknesses 
were  a  lack  of  a  Gantt  chart  capability  and  a  marginal  export  capability.  Expect 
outstanding  customer  support  from  i2.  It  is  no  coincidence  that  Analyst  Notebook  scored  a 
perfect  score  for  confidence.  The  i2  team  was  readily  accessible,  ensured  questions  were 
answered,  and  were  very  patient  with  the  study  team’s  questions  and  needs. 

Recommended  for  any  size  shop  that  can  afford  the  price  tag  and  wants  to  get  rolling 
quickly  w  ith  their  analytical  efforts,  and  particularly  valuable  if  the  shop  has  access  to 
various  data  sources. 

2.2.6  Pen-Link  Analyst 

Pen-Link  Analyst  by  Pen-Link  LTD  was  a  shareware  and  commercial  product 
developed  primarily  for  law  enforcement  working  with  warrant  access  to  telephone 
conversations.  The  tool  places  phone  calls  on  a  time  line  to  assist  law  enforcement  officers 
investigating  and  building  their  cases.  It  scored  an  overall  4.2  placing  it  in  last  place.  It 


:  PR  News  win*.  *  i2  Inc.  Wins  $2M  Federal  Contract  from  FBI”  31  July  2001. 

*  i2  Analyst  Notebook  f>.  Pre-release  Symposium  The  Key  Bridge  Murriot.  Rossi yn,  Virginia.  7  August  2002. 

79  PR  Newswinc  ‘Wai  on  Drugs.  Terrorism  to  Benefit  from  i2  Inc.’s  Groundbreaking  Telephone  Call  Pattern  Analysis 
Software.”  Financial  News  section  13  March  2002. 
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was  below  average  on  meeting  requirements  with  a  score  of  3.8.  The  tool  was  rich  in  user 
features,  including  a  PDA  version  of  the  tool.  The  only  weaknesses  known  was  the  lack  of 
a  Gantt  chart  capability.  The  tool  ranked  second  to  List  on  the  confidence  scale,  and  it  was 
unclear  if  it  supported  a  rich  data  import/export  capability,  Post-it  *  note  capability, 
multiple  threads,  ability  to  manipulate  events  on  the  timeline,  and  the  ability  to  have 
several  scales  on  the  same  time  line.  Customer  support  is  unknown  as  well,  except  it  was 
not  available  on  the  weekend.  Recommend  for  further  consideration  by  shops  that  are  law 
enforcement  oriented. 

2.2.7  CaseTrak  III 

CaseTrak  III  developed  by  Badge  1022  Software  as  shareware.  It  w  as  designed  for  the 
law  enforcement  community  users  on  a  tight  budget.  It  scored  an  overall  6.3  placing  it. 
surprisingly,  above  two  other  commercial  products.  It  was  below  average  on  meeting 
requirements  with  a  score  of  3.1.  Its  strength  w  as  in  data  references  and  query  capabilities. 
Its  weaknesses  were  that  it  was  still  a  DOS  based  program,  had  poor  data  import/export 
capabilities,  and  no  graphical  interface.  Customer  support  would  be  problematic  for  Badge 
1022  Software,  which  was  one  person  However,  w-e  did  receive  customer  support  on  a 
holiday  weekend'0.  Not  recommended  at  this  time  except  for  an  occasional  analyst  who 
might  find  it  useful.  Badge  1022  Software  is  considering  redevelopment  on  Windows'1.  If 
a  Windows  version  is  released  it  is  recommended  analytical  shops  revisit  this  practical 
tool. 

2.2.8  VisuaLinks 

VisuaLinks  developed  by  Visual  Analytics  was  offered  as  a  commercial  product  for 
an  undisclosed  price  tag.  The  vendor  supports  the  law  enforcement,  intelligence’2, 
corporate1'  and  government-*4  communities.  The  tool  scored  an  overall  5.4  placing  it 
second  to  last.  However,  it  scored  a  9.2  on  its  vendor-declared  ability  to  meet 
requirements.  VisuaLinks  was  the  link  analysis  and  visualization  software  used  for  Joint 
Intelligence  Virtual  Architecture  at  the  Defense  Intelligence  Agency'5.  The  low  overall 
score  was  a  result  of  the  study  team’s  inability  to  evaluate  the  tool.  Therefore,  the 
confidence  score  was  1.5.  The  study  team  w'as  able  to  verify  from  news  sources  that  it  did 
have  an  XML  interface  and  ability  to  pull  data  dynamically1' .  Expect  frustrating  customer 
support  for  this  tool.  Although  we  inquired  late  with  Visual  Analytics,  there  was  sufficient 
time  to  handle  our  request  to  evaluate  the  tool  anil  respond  to  the  requirements'7.  We  did 


Larry  Rite.  Badge  1022  Software.  ‘’Re:  CaseTrak  111"  E-mail  in  John  Wiglc.  31  August  2002. 

1  See  previous  footnote. 

PR  Newswirc.  “VisuaLinks  Chosen  as  Link  Analysis  Tt>ol  lor  Major  L)l A  and  DISA  projects**.  Financial  News  25  June 
2002. 

u  ^  Newswirc.  “Visual  Analytics  Expands  International  Base  into  Asia.**.  Financial  News  section  5  December  2(H) I . 

34  PR  Newswirc.  “Visual  Analytics  Inc  Licenses  Visual  Data  Mining  Software  for  Use  in  Counter-Drug  Operations**. 
Financial  News  section.  8  November  2000 

33  PR  Newswirc.  “VisuaLinks  Selected  as  Best -of- Breed  System  for  Large-Scale  U.S.  Intelligence  Program"  Financial  News 
section.  14  February  2002. 

Kyle  Balluck.  “Feds  to  Use  VAI  App  in  Drug  War"  Newshvios  x  November  2(KX» 

37  John  Wigle  “Timeline  analysis  capabilities..."  E-mail  to  Lynce  Weston  2 1  August  2002. 
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receive  a  response  to  the  requirements",  hut  we  were  unable  to  get  the  appropriate  license 
keys  and  login  information11  to  begin  out  evaluation  of  the  tool  within  a  week. 
Recommend  analytical  shops  of  any  size  consider  this  tool  based  on  its  requirement  rating, 
but  carefully  test  drive  the  tool  and  the  customer  support  before  investing  your  business 
processes  in  the  tool. 


,x  L.yncc  Weston.  Visual  Analytics.  **FW:  Timeline  uruily*K capabilities..."  E-mail  to  John  Wigle.  2ft  August  2002. 

Lynee  Weston.  Visual  Analytics.  “RE:  Ncu  Client  Profile  lor  jwigle@mitre.org**  L-mail  to  John  Wigle  29  August  2002. 
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Table  2  -  Tool  Comparison  Chart  -  Part  l  of  2 
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Table  2  -  Tool  Comparison  Chari  -  Part  2  of  2 
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POP 

Vlaio 

nc 

not  reported 

no 

yes' 

HTML  (web) 

rv> 
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not  reported 

no 

vws  ’ 

>»w 

not  report  oa 

nc 

tree1 

Qujxjt  la  iwoa  PMI«  dowees 

Plotter  Capable 

yet 

not  reponeo 

no 

MS' 

Wukive  and  imr  fnendv  rtertwx 
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no 
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2.3  Tools  Not  Investigated 

•  File  List  by  Forensics,  Inc 

•  Show  File  by  Forensic's,  Inc 

•  TlSU  WIZARD  BY  WIZDOM  ANALYSIS  TOOLS 

•  Flow  wizard  by  WIZdom  Analysis  Tools 

•  Experience Ware1  M  by  PRAJA.  Inc. 

•  Egress  Pro  by  S  i  mC o  Consu  1  ( i  n  c 

•  WinForce  by  winForce  Technologies 

•  C-/nsight  by  MetaEdge  Corporation 

•  DB2  Intelligence  Miner  by  IBM  Corporation 

These  tools  were  briefly  reviewed  because  the  vendor  did  not  respond  to  inquiries,  or 
the  study  team  concluded  their  purpose  was  disparate  from  intelligence  analysis,  or  both. 
However,  the  study  team  felt  it  was  important  for  a  minor  sidebar  discussion  about  these 
tools  because  they  had  displayed  capabilities  in  dealing  with  events  referenced  by  time  and 
date,  and  the  study  team  freely  admitted  their  limited  ability  to  understand  the  absolute 
needs  of  intelligence  analysis.  Therefore,  these  tools  were  included  in  this  report  to  help 
those  who  may  be  interested  in  learning  more  about  them. 
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2.3.1  FileList  and  ShowFile 

FileList  and  ShowFile  were  products  of  Forensics.  Inc.  They  created  a  time  line 
analysis  of  computer  file  dates  and  times  from  one  or  more  compute]  hard  disk  drives  and 
floppy  diskettes.  Time  lines  could  be  created  based  on  tile  access  dates,  file  creation  dates, 
file  modification  dates,  and  activity  associated  with  deleting  files.  Identification  of 
computer  activity  on  weekends  and  outside  of  normal  working  hours  could  be  put  into  a 
separate  time  line.  ShowFile  sorts,  analyzes,  and  \  iews  database  output  created  by 
F-ileList.  Time  line  analysis  of  computer  data  would  be  extremely  helpful  in  investigations 
and  computer  security  review  s. 

2.3.2  Flow  WIZard  and  Time  WIZard 

The  WIZdom  Analysis  Tool  Suite  had  two  useful  tools  called  the  Flow  WIZard  and 
Time  WIZard  tools.  Flow  WIZard  generated  process  flow  models  for  process  mapping. 
Time  WIZard  was  a  project  management  tool  utilizing  time  lines.  It  would  lie  used  to 
reverse  engineer  a  major  development  project  using  a  Gantt  chart  process  which  was  one 
of  the  high-level  requirements  identified. 

2.3.3  ExperienceWare 

Experience  Ware*™  was  a  rapidly  deployable  business  activity  monitoring  solution 
from  PRAJA.  Inc40.  Their  customers  included  Zurich  Financial  Services,  FOX.  University 
of  Chicago,  CBS,  Yahoo.  General  Motors,  and  Henry  Ford  Health  System.  The  tool 
allowed  companies  to  monitor  business  processes  by  plotting  activity  three  dimensionally 
using  location,  process,  and  time.  One  could  see  (hat  this  could  be  used  as  a  time  line 
analysis  tool,  but  the  tool  would  probably  be  used  irregularly  from  its  design  and  that 
could  become  problematic. 


2.3.4  EgressPro 

EgressPro  was  a  fire  engineering  simulator  developed  by  SimCo  Consulting,  and  was 
sold  for  $150  per  desktop.  The  simulator  develops  time  lines  and  movements  of  escape 
from  fires  based  on  established  fire  protection  engineering  theory.  Was  designed  to  assist 
architects  and  fire  protection  engineers  in  developing  models  to  predict  egress  times  for  a 
given  room,  corridor  or  stairwell41.  The  tool  runs  on  a  Windows  95/98  or  higher  operating 
system.  The  tool  has  no  direct  application  to  intelligence  analysis  that  the  study  team  was 
aware  of,  but  for  the  price  it  may  be  worthy  of  looking  at  if  your  analysis  includes  times  to 
egress  from  locations  that  are  not  necessarily  on  lire. 


40  ExperienceWare’  ' Advertisement.  Internet.  31  August  2002. 

**  £wnputer  Models  tor  Fire  and  Smoke.  Society  of  Fire  Protection  Engineers  International  Survey  of  Computer  Models  for 
Fire  and  Smoke.  <http://www.firemtxJelsurvey.com/pdf/ligre*sPro_2<)0l.pdf>  27  July  2002. 
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2.3.5  winForce 

winForce  is  ;i  commercial  tool  offered  by  winForce  Technologies  of  Scottsdale,  Arizona 
for  S595  per  desktop4*.  It  was  a  full  featured  case  management  system  designed  for 
lawyers  and  prosecutors.  The  tool  included  the  ability  to  place  data  entered  into  the  system 
automatically  onto  a  timeline  to  present  information  about  the  criminal  offense,  the 
investigation  and  the  evidence4'.  The  vendor  was  unresponsive  to  the  study  team’s  request 
for  information.  Their  website  is  http://www.blueknighttech.com/  and  includes  a 
download  section,  but  a  trial  version  had  to  be  mailed  out. 

2.3.6  C-Insight 

C-lnsight  a  commercial  tool  of  fered  by  MetaEdge  Corporation  of  Silicon  Valley  is 
designed  for  customer  intelligence  for  e-commerce  business.  C-Insight  allows  businesses 
to  review  customer  traffic  in  a  time  line  fashion.44 

2.3.7  DB2  Intelligent  Miner 

DB2  Intelligent  Miner  offered  by  IBM  includes  time-sequencing  functions.  Although 
not  a  stand-alone  tool  (requires  Business  Intelligence  Platform4*),  it  could  be  incorporated 
into  existing  customer  solutions.  In  a  news  article,  a  biopharniaceutical  company  used 
DB2  Intelligent  Miner  to  determine  the  progressions  of  diseases  with  timelines4*. 


1  “Welcome  lo  winForce  Technologies".  winForce  Home  Puce  <http://www.hlueknighttech.com/>  31  July  2002. 

J  '  Michael  Rogers.  Attorney.  “'WinForce  Rnniuct  Review".  Lceal  Tech.  November  2001.  Vol.  19.  No  8.  pg.  6 
u  PR  Ncwswire.  *  MetaEdge ’s  C-Insight  Provides  State-Of-The-Art  Customer  Intelligence  for  E-Enterprises"  12  September 
2000,  Financial  News  section. 

45  "IBM  Software  :  Database  and  Data  Management  :  Intelligent  Miner  Family  :  Overview”  DB2  Intelligent  Miner  web  page. 
<http://www-3.ibm.com/sofiware/dula/iminer/>  17  August  2(X)2. 

l"  PR  Ncwswire.  “AxCell  Biosciences  Explores  Pathways  to  disease  with  IBM  Database  Technology"  16  August  2002. 
Financial  News  section. 
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III. SUMMARY 

3.1  Recommendations 

Based  upon  ihe  given  requirements,  the  team's  research,  and  the  information  provided 
by  the  vendors,  our  recommendation  was  i2*s  Analyst  Notebook  Version  6.  It  was  the 
most  intuitive,  was  verified  to  have  had  most  of  the  high-level  requirements,  and  had 
helpful  documentation.  The  i2  staff  was  also  very  helpful  and  quick  to  respond  questions 
and  support.  Unfortunately  it  was  not  due  out  until  the  end  of  the  2002  calendar  year. 

The  other  tool  to  examine  closely  would  be  VisuaLinks  by  Visual  Analytics. 

And  finally,  watch  for  a  Windows  release  of  CaseTrak  111  by  Badge  1022  Software. 
The  tool  remained  promising,  and  showed  well  lor  a  shareware  product. 

3.2  Conclusion 

There  were  some  powerful  tools  available  for  use  by  the  1C  on  the  commercial  market, 
but  no  holy  grail.47  None  of  them  met  all  the  high-level  requirements  identified  by  the 
intelligence  analysts.  Of  the  evaluated  tools,  a  few  met  most  of  the  high-level 
requirements  for  time  line  analysis,  and  most  of  the  tools  met  at  least  hall  the  high-level 
requirements.  This  disparity  of  capabilities  illustrated  the  time  line  analysis  market  was 
still  maturating.  There  were  still  too  few  tools  available.  The  need  for  better  time  line 
analysis  tools  still  existed  for  increasing  the  MITRE  Analyst  Tool  Shed  product  line  and  to 
combat  the  Asymmetric  Order  of  Battle  (ASOB)  problem. 

It  is  possible  that  the  time  line  analysis  market,  and  in  general  the  automated 
intelligence  analysis  markets,  will  continue  to  evolve  into  a  richer  market  of  tools.  Newer 
products  could  be  introduced  and  further  development  of  existing  tools  will  only  improve 
the  availability  and  selection  of  time  line  analysis  tools.  Identification  of  analytical 
requirements  of  a  customer  will  remain  essential  to  determining  which  tool  will  work  best 
for  a  given  environment.  And  it  may  become  necessary  to  customize  existing  commercial 
software  through  application  program  interfaces  to  satisfy  the  needs  of  intelligence 
analysts. 

This  market  survey  may  serve  as  a  baseline  for  future  efforts  at  analyzing  the  time  line 
analysis  market.  In  the  event  that  the  time  line  analysis  market  does  not  grow,  it  may  be 
practical  for  MITRE  to  develop  customized  API  plug-ins  for  commercial  software  or  to 
develop  a  tool  to  match  the  requirements  of  the  intelligence  analyst  community. 


3.3  Contact  Information 

You  may  direct  your  questions  to  the  study  team  at  the  following  phone  numbers  and 
email  addresses  listed  in  the  table  below: 

Table  3  -  Study  Team  Contact  Information 


4  Jim  Bumciti.  ‘Timeline  Analysis  Tools  Evaluations.*  E-msul  lo  John  Wigle.  31  May  2iX)2. 
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Senior  Advisor  Jerry  Cogie 

Lead  Researcher  John  Wigle 

Researcher  Jennifer  Websiei 


(703)883-6277 
(703)  883-1277 
(703)  883-6039 


jcogle@mitre.org 

jwigle@miire.org 

jenw@mitre.org 
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IV.  Glossary  of  Terms  and  Phrases 

API  -  An  acronym  for  Application  Program  interface.  An  interface  designed  into  computer  programs  that  allow 
third  parties  to  develop  external  software  that  works  with  the  application  like  if  it  was  built  into  the  application 
itself.  Typically,  the  software  modules  designed  to  work  with  applications  through  an  API  is  called  a  plug-in, 
because  it  is  imagined  to  plug  into  the  tool.  See  Plug  in 

ASOB  -  An  acronym  for  ASymmetric  Order  of  Battle. 

Asymmetric  Order  of  Battle  -  The  steps  an  entity  goes  through  to  read)  itself  for  an  irregular  (asymmetric) 
offensive  or  defensive  position.  The  acronym  for  this  phrase  is  ASOB  Compare  with  Order  of  Battle  and 
Signature. 

DTD  —  An  acronym  for  Data  Table  Description.  A  DTD  is  a  document  used  to  describe  the  data  elements  in  a 
data  file,  so  a  third  party  will  understand  how  the  data  is  organized  and  what  data  is  stored 

Gantt  Chart  -  A  specific  type  of  chan  used  in  program  management  that  is  named  after  an  American  scientist 
who  invented  it.  The  chart  uses  milestones  to  denote  the  beginning  and  ending  of  a  specific  task  w  ith  a  horizontal 
line  drawn  between  them.  The  length  of  the  horizontal  line  represents  the  time  it  takes  to  complete  the  task.  The 
chart  comprises  all  the  tasks  needed  to  complete  a  project  spread  out  over  a  time  line  to  illustrate  how  tlte  long 
the  project  may  take  to  complete.  Independent  parallel  lasks  can  he  shown  above  or  below  other  tasks,  and  tasks 
dependent  upon  another  task  being  finished  first  follow  alter  the  first  task.  The  chart  was  designed  to  show  the 
dependencies  and  the  critical  points  in  the  project  that  need  to  he  managed  closely 

GIF  -  An  acronym  for  Graphic  Interchange  Format.  A  photographic  file  format  developed  b\  CompuServes  a 
Internet  Service  Provider  like  American  On-Line,  to  allow  their  users  to  exchange  photos.  A  common  graphic 
format  used  on  the  Internet  Compare  with  JPEG  and  SGML. 

HTML  -  An  acronym  for  HyperText  Markup  Language.  HTML  is  an  Internet  standard  used  to  exchange 
document  information  semantically,  leaving  the  client  application,  called  a  browser,  to  determine  the  proper  w>ay 
to  display  a  document 

Informix  -  A  large-scale  commercial  database  company,  w  ith  a  database  application  by  the  same  name.  Compare 
with  Ingress.  Microsoft  Access,  Microsoft  SQL  Server.  Oracle,  and  Sybase. 

Ingress  -  A  medium  scale  commercial  database  application  available  from  a  company  called  Computer 
Associates.  Compare  with  Informix,  Microsoft  Access.  Microsoft  SQL  Server,  Oracle,  ami  Sybase. 

Intelligence  Community  -  The  agencies  and  services  of  a  government  that  conduct  intelligence  activities  In  the 
United  States  the  Intelligence  Community  is  defined  by  membership  on  the  National  Foreign  Intelligence  Board 
Currently  there  are  fourteen  agencies  and  departments  having  membership  on  this  board  They  are  the  Federal 
Bureau  of  Investigation,  the  Central  Intelligence  Agency,  the  Army,  the  Navy,  the  Air  Force,  the  Marines,  the 
U.S.  Coast  Guard,  the  Department  of  Treasury,  the  Department  of  Energy,  the  Department  ol  State,  the  National 
Security  Agency,  the  Defense  Intelligence  Agency,  the  National  Image  and  Mapping  Agent),  and  the  National 
Reconnaissance  Office. 

JPEG  -  An  acronym  for  Jet  Propulsion  Exchange  Graphic.  A  photographic  file  format  developed  by  the  Jet 
Propulsion  Labs  to  allow  photos  from  space  to  be  shared  with  scientists  over  the  Internet.  A  very  common 
standard  used  on  the  Internet.  Compare  with  GIF  and  SGML. 

Microsoft  Access  -  A  commercial  software  database  from  Microsoft  Corporation  that  typically  works  only  for 
one  user  or  small  business  applications.  Compare  with  Informix.  Ingress.  Microsoft  SQL  Server,  Oracle,  and 
Sybase. 

Microsoft  SQL  Server  -  A  commercial  software  database  from  Microsoft  Corporation  that  is  designed  for  larger 
scale  use  than  Access.  Compare  with  Informix,  Ingress.  Microsoft  Access.  Oracle,  and  Sybase 

ODBC  -  An  acronym  for  Open  DataBase  Connectivity.  An  industry  standard  used  to  allow  third  party  software 
to  connect  to  any  ODBC-compliant  database.  Compare  with  XML. 
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Oracle  -  A  commercial  database  company.  with  a  database  application  by  the  same  name  designed  for  large, 
medium  and  small-scale  usage.  Compare  with  Informix.  Ingress,  Microsoft  Access.  Microsoft  SQL  Server,  and 
Sybase. 

Order  o)  Battle  ~  The  defined  steps  a  country  or  entity  goes  through  to  ready  themselves  for  offensive  attack  or 
defensive  position.  Compare  with  Asvmmetnc  Order  of  Battle  and  Signature. 

PDF  -  A  file  extension.  Files  ending  with  ‘.pdf  are  written  in  Adobe  Acrobat  format.  Adobe  offers  a  free  file 
viewer  called  Acrobat  to  allow  users  to  read  and  print  these  files. 

Plotter  —  A  printing  device  used  to  print  very  large  graphics,  like  posters,  blue  prints,  charts,  etc.  D-Size  plotters 
handle  paper  about  three  or  tour  feet  wide 

Plug-in  —  Software  that  does  not  w?ork  alone  as  an  application,  but  plugs  into  a  third  party  software  application, 
and  provides  that  third-party  application  with  enhanced  capabilities  it  natively  does  not  have.  See  API. 

SGML  -  An  acronym  for  Standardized  Graphic  Markup  Language.  SGML  is  a  standard  from  which  other  mark¬ 
up  languages  evolved.  For  instance.  HTML  is  a  sub  standard  of  SGML  SGML  is  used  to  describe  graphics  in  a 
standardize  way  so  they  can  be  freely  shared  Compare  with  JPFG  and  GIF. 

Signature  -  The  unique  characteristics  that  make  up  an  intelligence  event.  For  instance,  il  a  person  buys  a  ski 
mask,  a  handgun,  paper,  pen.  and  steals  a  car  is  probably  preparing  to  rob  a  bank  The  steps  used  to  prepure  tor 
the  robbery  is  called  an  order  of  battle,  and  the  unique  steps  used  to  identify  it’s  a  bank  robbery  is  called  the 
signature.  Compare  w  ith  Order  of  Battle  and  Asymmetric  Order  of  Battle 

Sybase  -  A  commercial  database  company,  w  ith  a  large-scale  database  application  by  the  same  name  Compare 
w'ith  Informix.  Ingress.  Microsoft  Access,  Microsoft  SQL  Server,  and  Oracle. 

Visio  -  A  Microsoft  software  product  name.  The  application  is  designed  to  draw  schematics,  charts  and  diagrams 

Wizard-like  Interface  -  A  common  interface  used  to  install  applications  on  Microsoft  Windows.  The  interface 
asks  the  user  a  series  of  questions.  Based  on  the  answers  given,  the  wizard  automates  the  process  to  assist  the 
user. 

XML  -  Acronym  for  extensible  Markup  Language.  An  Internet  standard  used  to  allow'  anyone  understanding 
XML  to  interpret  data  in  any  source.  Compare  with  ODBC. 
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100  Flags/prohibits  cycles  100  Flags/prohibits  cycles 

50  Cycles  only;  autonormalizes  50  Cycles  only;  autonormalizes 

0  Automatically  normalizes  0  Automatically  normalizes 
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NAME 

ORGANIZATION 

POSITION 

PHONE 

EMAIL 

Greg  Jannarone 

NAIC/BPB 

Primary  Customer 

937-656-1055 

auoust.iannarone@wDafb.af.mil 

Eric  Braeden 

NAIC/BPB 

NASIC  Lead 

937-910-6490 

Braeden@sbcqlobal.net 

Ken  Hammerte 

NAIC/BPB 

NASIC  Analyst 

937-656-1055 

kenneth.hammerle@wDafb.af.mil 

Gil  Kuperman 

AFRL/HECA 

Contract  Tech  Lead 

937-255-3727 

ailbert.kuDerman@wDafb.af.mil 

Capt  Tim  Gameros 

AFRL/HECA 

PSYOP  Researcher 

937-255-4046 

timothv.qameros@wDafb.af.mil 

Don  Monk 

AFRL/HECA 

COAST  (IWPC) 

donald.monk@wDafb.af.mil 

Dave  Morton 

AFRL 

M&S  Lead 

937-255-9349 

david.morton@wDafb.  af.mil 

Dr.  Jon  Pfautz 

Charles  River  Analytics 

OCCAM 

617-491-3474  x541 

iDfautz@cra.com 

Karen  Harper 

Charles  River  Analytics 

Sample,  Grade 

617-491-3474  x533 

kharDer@cra.com 

Dr.  Mark  Sheehan 

AFRL/HE 

CRA  SBIR  PM 

937-255-8806 

Bill  Curtice 

ASC/ENM 

M&S  Division  Chief 

937-904-4409 

william.curtice@wDafb.af.mil 

Joe  Reiman 

ASC/ENM 

M&S  Division 

937-904-4553 

Larry  Daniel 

SRA 

Project  Manager 

937-910-6433 

larrv  daniel@sra.com 

Mike  Zywien 

SRA 

Tech  Lead 

937-910-6425 

michael  zvwien@ sra.com 

Elisabeth  Fitzhugh 

SRA 

Human  Factors 

937-910-411 

elisabeth  fitzhuah@sra.com 

Aaron  Bryant 

SRA 

Intelligence 

937-910-6426 

aaron  brvant@sra.com 

Ratna  Bearavolu 

SRA 

Software 

937-910-6455 

ratna  bearavolu@sra.com 

Rick  Raftery 

SRA 

HUMINT 

703-684-2347 

rick  rafterv@sra.com 

Travis  Lynch 

JWAC 

JEMNA 

540-653-3757 

Phil  Desmans 

JWAC 

JEMNA 

540-653-5757 

Bob  Molepske 

JWAC  -  Raytheon 

JEMNA  -  Toffler 

703-413-5337 

Ken  Moak 

AT&T 

BA&PS 

937-320-4571 

kmoak@att.com 

Lt  Col  Bob  Rushing 

JIOC 

BA&PS 

210-977-2484 

William  Swart 

JIOC 

william.swart@iioc.osis.qov 

Mike  Millen 

JIOC 

210-977-2483 

Larry  Willis 

DARPA 

(703)  696-7448 

lwillis@daroa.mil 

Joe  Schaff 

NAVAIR 

(301)  904-3759 

SchaffJB@navair.naw.mil 

Vincente  Garcia 

New  Mexico  State  Univ. 

vcoarcia@Dsl.nmsu.edu 

Ray  Buettner 

Naval  Postgraduate  School 

(831)  656-3387 

rrbuettn@nDS.naw.mil 

Dr.  Julie  Rosen 

SAIC 

SIAM 

703-676-7354 

iulie.a.rosen@saic.com 

Gary  Fauss 

MRC 

PRFECT 

505-301-2591 

oarv.fauss@kirtland.af.mil 

Col  Jack  Summe 

Mrs.  Graham  (Sec) 

USA  4th  POG  -  Ft  Bragg 

Group  CC 

910-432-6392 

Col  Alan  Snyder 

AIA/DO-2 

AF  PSYOP  PMO  IPT 

210-977-2116 

Frank  Goldstein 

DO-2 

210-579-3940 

Glenn  Kinder 

DO-2 

XP  Chief  of  PSYOP 

210-977-2116 

qlenn.kinder@lackland.af.mil 

Lt  Col  Doug  Jaquish 

ASC/RAB 

Commando  Solo  PM 

255-2211 

douo.iaouish@wDafb.af.mil 

Ron  Swartzwelder 

NSA 

ASC  Rep  @  RA 

255-4326 

ronald.swartzwelder@wDafb.af.mil 

Cindy  Farkus 

NSA 

301-688-7111 

Bert  Head 

NSA 

Dep  Ops  Rsrch  M&S 

301-688-2857 

Bill  Ryder 

NSA 

Tech  Lead  Ops  Rsrch  M&S 

301-688-4648 

whrvder@nsa.aov 

Lt  Col  Francine  Goode 

NSA 

Ops  Rsrch  M&S 

301-688-2869 

Dr  Tom  Smith 

NSA 

DPLF  Tool  in  OR  M&S 

301-688-2851 

Jeanne  Jones 

DIA 

AFMC  rep  (@  NAIC) 

257-3725 

Col  Jeff  Buckmelter 

SOCOM 

Mr.  IW  for  SOCOM 

Tim  Menke 

SIMAF 

ASC  M&S 

937-255-1276 

Randy  Levine 

SIMAF 

ASC  M&S 

937-255-0672 

Brooke  McNally 

SIMAF 

ASC  M&S 

937-255-0625 

Nicolas  Rummelt 

Eglin  AFB,  FL 

Ex  ASC  M&S  Expert 

850-882-3910x2320 
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Accessibility  -  Measure  of  whether  a  given  target  of  interest  can  be  reached 
(physically  or  by  other  means)  to  exploit  a  vulnerability 

Analytical  Hierarchy  Process  -  A  multi-criteria  decision  analysis  technique  that 
uses  a  prioritized  hierarchy  of  criteria  to  aid  a  decision  process 

Alterables  -  Aspects  of  a  given  problem  that  can  be  manipulated,  traded,  or 
changed  to  help  drive  a  solution 

Bayes’  Theorem  -  The  posterior  probability  distribution  for  a  given  set  of 
random  variables  can  be  calculated  given  an  exact  value  for  other  variables 

Bayesian/Belief/Causal/Probability  Nets  -  Networks  (nodes  and  links)  used  to 
represent  dependencies  between  random  variables;  nodes  (parent  and  child)  are 
associated  with  conditional  probability  tables,  links  represent  dependencies 
between  nodes 

Commercial  Off-The-Shelf  (COTS)  -  Technologies,  products,  and  services  that 
can  be  purchased  directly  from  commercial  vendors 

Courses  of  Action  (COA)  -  Alternative  ways  to  approach  and  address  a  given 
problem 

Criteria  Model  -  A  hierarchal  structure  with  a  top  level  goal,  criteria  and  sub¬ 
criteria  that  affect  the  goal,  and  a  bottom  level  set  of  alternatives;  each  element 
of  the  model  is  assigned  weights,  and  the  weights  determine  the  influence  one 
element  of  the  model  has  on  another 

Data  Mining  -  Methods  and  technologies  associated  with  finding  a  desired  piece 
of  information  in  accessible  data  (text,  video,  audio,  etc.)  files 

Decision  Node  Support  -  Support  for  nodes  that  can  be  used  to  represent  the 
decisions  and  the  alternatives  that  the  user  is  faced  with. 

Decision  Support  System  (DSS)  -  Computer-aided  tools  that  emulate  the 
reasoning  process  of  a  human  expert  making  decisions 

Decision  Trees  -  Chronological  model  of  a  problem  depicted  with  roots  and 
branches;  roots  are  decision  nodes,  branches  represent  various  possibilities  of 
the  decision  node  (branches  lead  to  decision  nodes,  chance  nodes,  or  result 
nodes) 

Decision/Utility  Theory  -  A  problem  solving  approach  that  strives  to  maximize  a 
given  utility  function 
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Government  Off-The-Shelf  (GOTS)  -  Technologies,  products,  and  services  that 
can  be  acquired  directly  from  other  government  agencies 

Inference  Algorithm  -  Algorithms  that  calculate  the  posterior  probability  of  the 
variables  (nodes)  in  the  Bayesian  net,  given  exact  values  of  some  variables 

Influence  Diagrams  -  Consists  of  a  Bayesian  Net,  a  Decision  Node  (alternative 
course  of  action),  and  a  Value  Node  (value  or  utility  of  a  given  outcome);  returns 
the  action  with  the  highest  utility 

Influence  Operations  -  Operations  focused  on  affecting  the  perceptions  and 
behaviors  of  people,  leaders,  groups,  or  entire  populations.  The  means  can  be 
physical  or  informational. 

Mixed  Graph  Support  -  Support  for  both  directed  and  undirected  links  in  a 
model 

Modeling  -  Process  of  representing  a  given  problem  or  reality  with  assigned 
values,  weights,  beliefs,  etc.  and  using  inference  techniques  to  answer  queries 

Psychological  Effects-Based  Operations  (PEBO)  -  the  deliberate  use  of  the 
USAF  core  operational  competencies  and  its  enabling  technologies  as  a 
psychological  instrument. 

Requirements  Trace  -  A  structured  methodology  to  tie  products,  services,  and 
technologies  and  components  of  products,  services,  and  technologies  to 
technical  and  operational  requirements 

Rules  of  Engagement  (ROE)  -  a  set  of  firm  rules,  usually  framed  by  national 
policy  and  implemented  by  the  Operational  Commander,  that  govern  the  conduct 
of  operations  in  a  given  theater 

Social  Network  Analysis  -  Social  network  analysis  [SNA]  is  the  mapping  and 
measuring  of  relationships  and  flows  between  people,  groups,  organizations, 
computers  or  other  information/knowledge  processing  entities.  The  nodes  in  the 
network  are  the  people  and  groups  while  the  links  show  relationships  or  flows 
between  the  nodes.  SNA  provides  both  a  visual  and  a  mathematical  analysis  of 
human  relationships.  (Definition  from  www.orgnet.com) 

Structure  Learning  -  Technique  that  automatically  generates  a  Bayesian  Net, 
given  data  (in  table  format) 

Susceptibility  -  A  predictable  path  to  exploit  a  given  vulnerability 
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Text  Mining  -  Methods  and  technologies  associated  with  finding  a  desired  piece 
of  information  in  accessible  text  files 

Virtual  Evidence  Support  -  Support  to  enter  evidence  into  the  Bayesian  net 
which  the  user  is  uncertain  about. 

Vulnerability  -  Characteristics  of  a  chosen  target  that  planners  and  analysts  can 
leverage  and  exploit 
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